alexion/dev
1
0 Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

build(rust): vendor googletest crate

Browse Source
This commit was merged in pull request #76.
This commit is contained in:
alexion committed 2026-09-19 14:53:22 -04:00
1 parent a6fe84190e
commit 6887b104d6
576 files changed
+284290 -2
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

third_party/crates/regex-automata-0.4.18/src/util/alphabet.rs
Vendored
+1139
View File
File diff suppressed because it is too large. Load diff
third_party/crates/regex-automata-0.4.18/src/util/captures.rs
Vendored
+2551
View File
File diff suppressed because it is too large. Load diff
third_party/crates/regex-automata-0.4.18/src/util/determinize/mod.rs
Vendored
+682
View File
@@ -0,0 +1,682 @@
/*!
This module contains types and routines for implementing determinization.
In this crate, there are at least two places where we implement
determinization: fully ahead-of-time compiled DFAs in the `dfa` module and
lazily compiled DFAs in the `hybrid` module. The stuff in this module
corresponds to the things that are in common between these implementations.
There are three broad things that our implementations of determinization have
in common, as defined by this module:
* The classification of start states. That is, whether we're dealing with
word boundaries, line boundaries, etc., is all the same. This also includes
the look-behind assertions that are satisfied by each starting state
classification.
* The representation of DFA states as sets of NFA states, including
convenience types for building these DFA states that are amenable to reusing
allocations.
* Routines for the "classical" parts of determinization: computing the
epsilon closure, tracking match states (with corresponding pattern IDs, since
we support multi-pattern finite automata) and, of course, computing the
transition function between states for units of input.
I did consider a couple of alternatives to this particular form of code reuse:
1. Don't do any code reuse. The problem here is that we *really* want both
forms of determinization to do exactly identical things when it comes to
their handling of NFA states. While our tests generally ensure this, the code
is tricky and large enough where not reusing code is a pretty big bummer.
2. Implement all of determinization once and make it generic over fully
compiled DFAs and lazily compiled DFAs. While I didn't actually try this
approach, my instinct is that it would be more complex than is needed here.
And the interface required would be pretty hairy. Instead, I think splitting
it into logical sub-components works better.
*/
use alloc::vec::Vec;
pub(crate) use self::state::{
State, StateBuilderEmpty, StateBuilderMatches, StateBuilderNFA,
};
use crate::{
nfa::thompson,
util::{
alphabet,
look::{Look, LookSet},
primitives::StateID,
search::MatchKind,
sparse_set::{SparseSet, SparseSets},
start::Start,
utf8,
},
};
mod state;
/// Compute the set of all reachable NFA states, including the full epsilon
/// closure, from a DFA state for a single unit of input. The set of reachable
/// states is returned as a `StateBuilderNFA`. The `StateBuilderNFA` returned
/// also includes any look-behind assertions satisfied by `unit`, in addition
/// to whether it is a match state. For multi-pattern DFAs, the builder will
/// also include the pattern IDs that match (in the order seen).
///
/// `nfa` must be able to resolve any NFA state in `state` and any NFA state
/// reachable via the epsilon closure of any NFA state in `state`. `sparses`
/// must have capacity equivalent to `nfa.len()`.
///
/// `match_kind` should correspond to the match semantics implemented by the
/// DFA being built. Generally speaking, for leftmost-first match semantics,
/// states that appear after the first NFA match state will not be included in
/// the `StateBuilderNFA` returned since they are impossible to visit.
///
/// `sparses` is used as scratch space for NFA traversal. Other than their
/// capacity requirements (detailed above), there are no requirements on what's
/// contained within them (if anything). Similarly, what's inside of them once
/// this routine returns is unspecified.
///
/// `stack` must have length 0. It is used as scratch space for depth first
/// traversal. After returning, it is guaranteed that `stack` will have length
/// 0.
///
/// `state` corresponds to the current DFA state on which one wants to compute
/// the transition for the input `unit`.
///
/// `empty_builder` corresponds to the builder allocation to use to produce a
/// complete `StateBuilderNFA` state. If the state is not needed (or is already
/// cached), then it can be cleared and reused without needing to create a new
/// `State`. The `StateBuilderNFA` state returned is final and ready to be
/// turned into a `State` if necessary.
pub(crate) fn next(
nfa: &thompson::NFA,
match_kind: MatchKind,
sparses: &mut SparseSets,
stack: &mut Vec<StateID>,
state: &State,
unit: alphabet::Unit,
empty_builder: StateBuilderEmpty,
) -> StateBuilderNFA {
sparses.clear();
// Whether the NFA is matched in reverse or not. We use this in some
// conditional logic for dealing with the exceptionally annoying CRLF-aware
// line anchors.
let rev = nfa.is_reverse();
// The look-around matcher that our NFA is configured with. We don't
// actually use it to match look-around assertions, but we do need its
// configuration for constructing states consistent with how it matches.
let lookm = nfa.look_matcher();
// Put the NFA state IDs into a sparse set in case we need to
// re-compute their epsilon closure.
//
// Doing this state shuffling is technically not necessary unless some
// kind of look-around is used in the DFA. Some ad hoc experiments
// suggested that avoiding this didn't lead to much of an improvement,
// but perhaps more rigorous experimentation should be done. And in
// particular, avoiding this check requires some light refactoring of
// the code below.
state.iter_nfa_state_ids(|nfa_id| {
sparses.set1.insert(nfa_id);
});
// Compute look-ahead assertions originating from the current state. Based
// on the input unit we're transitioning over, some additional set of
// assertions may be true. Thus, we re-compute this state's epsilon closure
// (but only if necessary). Notably, when we build a DFA state initially,
// we don't enable any look-ahead assertions because we don't know whether
// they're true or not at that point.
if !state.look_need().is_empty() {
// Add look-ahead assertions that are now true based on the current
// input unit.
let mut look_have = state.look_have();
match unit.as_u8() {
Some(b'\r') => {
if !rev || !state.is_half_crlf() {
look_have = look_have.insert(Look::EndCRLF);
}
}
Some(b'\n') => {
if rev || !state.is_half_crlf() {
look_have = look_have.insert(Look::EndCRLF);
}
}
Some(_) => {}
None => {
look_have = look_have
.insert(Look::End)
.insert(Look::EndLF)
.insert(Look::EndCRLF);
}
}
if unit.is_byte(lookm.get_line_terminator()) {
look_have = look_have.insert(Look::EndLF);
}
if state.is_half_crlf()
&& ((rev && !unit.is_byte(b'\r'))
|| (!rev && !unit.is_byte(b'\n')))
{
look_have = look_have.insert(Look::StartCRLF);
}
if state.is_from_word() == unit.is_word_byte() {
look_have = look_have
.insert(Look::WordAsciiNegate)
.insert(Look::WordUnicodeNegate);
} else {
look_have =
look_have.insert(Look::WordAscii).insert(Look::WordUnicode);
}
if !unit.is_word_byte() {
look_have = look_have
.insert(Look::WordEndHalfAscii)
.insert(Look::WordEndHalfUnicode);
}
if state.is_from_word() && !unit.is_word_byte() {
look_have = look_have
.insert(Look::WordEndAscii)
.insert(Look::WordEndUnicode);
} else if !state.is_from_word() && unit.is_word_byte() {
look_have = look_have
.insert(Look::WordStartAscii)
.insert(Look::WordStartUnicode);
}
// If we have new assertions satisfied that are among the set of
// assertions that exist in this state (that is, just because we added
// an EndLF assertion above doesn't mean there is an EndLF conditional
// epsilon transition in this state), then we re-compute this state's
// epsilon closure using the updated set of assertions.
//
// Note that since our DFA states omit unconditional epsilon
// transitions, this check is necessary for correctness. If we re-did
// the epsilon closure below needlessly, it could change based on the
// fact that we omitted epsilon states originally.
if !look_have
.subtract(state.look_have())
.intersect(state.look_need())
.is_empty()
{
for nfa_id in sparses.set1.iter() {
epsilon_closure(
nfa,
nfa_id,
look_have,
stack,
&mut sparses.set2,
);
}
sparses.swap();
sparses.set2.clear();
}
}
// Convert our empty builder into one that can record assertions and match
// pattern IDs.
let mut builder = empty_builder.into_matches();
// Set whether the StartLF look-behind assertion is true for this
// transition or not. The look-behind assertion for ASCII word boundaries
// is handled below.
if nfa.look_set_any().contains_anchor_line()
&& unit.is_byte(lookm.get_line_terminator())
{
// Why only handle StartLF here and not Start? That's because Start
// can only impact the starting state, which is special cased in
// start state handling.
builder.set_look_have(|have| have.insert(Look::StartLF));
}
// We also need to add StartCRLF to our assertions too, if we can. This
// is unfortunately a bit more complicated, because it depends on the
// direction of the search. In the forward direction, ^ matches after a
// \n, but in the reverse direction, ^ only matches after a \r. (This is
// further complicated by the fact that reverse a regex means changing a ^
// to a $ and vice versa.)
if nfa.look_set_any().contains_anchor_crlf()
&& ((rev && unit.is_byte(b'\r')) || (!rev && unit.is_byte(b'\n')))
{
builder.set_look_have(|have| have.insert(Look::StartCRLF));
}
// And also for the start-half word boundary assertions. As long as the
// look-behind byte is not a word char, then the assertions are satisfied.
if nfa.look_set_any().contains_word() && !unit.is_word_byte() {
builder.set_look_have(|have| {
have.insert(Look::WordStartHalfAscii)
.insert(Look::WordStartHalfUnicode)
});
}
for nfa_id in sparses.set1.iter() {
match *nfa.state(nfa_id) {
thompson::State::Union { .. }
| thompson::State::BinaryUnion { .. }
| thompson::State::Fail
| thompson::State::Look { .. }
| thompson::State::Capture { .. } => {}
thompson::State::Match { pattern_id } => {
// Notice here that we are calling the NEW state a match
// state if the OLD state we are transitioning from
// contains an NFA match state. This is precisely how we
// delay all matches by one byte and also what therefore
// guarantees that starting states cannot be match states.
//
// If we didn't delay matches by one byte, then whether
// a DFA is a matching state or not would be determined
// by whether one of its own constituent NFA states
// was a match state. (And that would be done in
// 'add_nfa_states'.)
//
// Also, 'add_match_pattern_id' requires that callers never
// pass duplicative pattern IDs. We do in fact uphold that
// guarantee here, but it's subtle. In particular, a Thompson
// NFA guarantees that each pattern has exactly one match
// state. Moreover, since we're iterating over the NFA state
// IDs in a set, we are guaranteed not to have any duplicative
// match states. Thus, it is impossible to add the same pattern
// ID more than once.
//
// N.B. We delay matches by 1 byte as a way to hack 1-byte
// look-around into DFA searches. This lets us support ^, $
// and ASCII-only \b. The delay is also why we need a special
// "end-of-input" (EOI) sentinel and why we need to follow the
// EOI sentinel at the end of every search. This final EOI
// transition is necessary to report matches found at the end
// of a haystack.
builder.add_match_pattern_id(pattern_id);
if !match_kind.continue_past_first_match() {
break;
}
}
thompson::State::ByteRange { ref trans } => {
if trans.matches_unit(unit) {
epsilon_closure(
nfa,
trans.next,
builder.look_have(),
stack,
&mut sparses.set2,
);
}
}
thompson::State::Sparse(ref sparse) => {
if let Some(next) = sparse.matches_unit(unit) {
epsilon_closure(
nfa,
next,
builder.look_have(),
stack,
&mut sparses.set2,
);
}
}
thompson::State::Dense(ref dense) => {
if let Some(next) = dense.matches_unit(unit) {
epsilon_closure(
nfa,
next,
builder.look_have(),
stack,
&mut sparses.set2,
);
}
}
}
}
// We only set the word byte if there's a word boundary look-around
// anywhere in this regex. Otherwise, there's no point in bloating the
// number of states if we don't have one.
//
// We also only set it when the state has a non-zero number of NFA states.
// Otherwise, we could wind up with states that *should* be DEAD states
// but are otherwise distinct from DEAD states because of this look-behind
// assertion being set. While this can't technically impact correctness *in
// theory*, it can create pathological DFAs that consume input until EOI or
// a quit byte is seen. Consuming until EOI isn't a correctness problem,
// but a (serious) perf problem. Hitting a quit byte, however, could be a
// correctness problem since it could cause search routines to report an
// error instead of a detected match once the quit state is entered. (The
// search routine could be made to be a bit smarter by reporting a match
// if one was detected once it enters a quit state (and indeed, the search
// routines in this crate do just that), but it seems better to prevent
// these things by construction if possible.)
if !sparses.set2.is_empty() {
if nfa.look_set_any().contains_word() && unit.is_word_byte() {
builder.set_is_from_word();
}
if nfa.look_set_any().contains_anchor_crlf()
&& ((rev && unit.is_byte(b'\n')) || (!rev && unit.is_byte(b'\r')))
{
builder.set_is_half_crlf();
}
}
let mut builder_nfa = builder.into_nfa();
add_nfa_states(nfa, &sparses.set2, &mut builder_nfa);
builder_nfa
}
/// Compute the epsilon closure for the given NFA state. The epsilon closure
/// consists of all NFA state IDs, including `start_nfa_id`, that can be
/// reached from `start_nfa_id` without consuming any input. These state IDs
/// are written to `set` in the order they are visited, but only if they are
/// not already in `set`. `start_nfa_id` must be a valid state ID for the NFA
/// given.
///
/// `look_have` consists of the satisfied assertions at the current
/// position. For conditional look-around epsilon transitions, these are
/// only followed if they are satisfied by `look_have`.
///
/// `stack` must have length 0. It is used as scratch space for depth first
/// traversal. After returning, it is guaranteed that `stack` will have length
/// 0.
pub(crate) fn epsilon_closure(
nfa: &thompson::NFA,
start_nfa_id: StateID,
look_have: LookSet,
stack: &mut Vec<StateID>,
set: &mut SparseSet,
) {
assert!(stack.is_empty());
// If this isn't an epsilon state, then the epsilon closure is always just
// itself, so there's no need to spin up the machinery below to handle it.
if !nfa.state(start_nfa_id).is_epsilon() {
set.insert(start_nfa_id);
return;
}
stack.push(start_nfa_id);
while let Some(mut id) = stack.pop() {
// In many cases, we can avoid stack operations when an NFA state only
// adds one new state to visit. In that case, we just set our ID to
// that state and mush on. We only use the stack when an NFA state
// introduces multiple new states to visit.
loop {
// Insert this NFA state, and if it's already in the set and thus
// already visited, then we can move on to the next one.
if !set.insert(id) {
break;
}
match *nfa.state(id) {
thompson::State::ByteRange { .. }
| thompson::State::Sparse { .. }
| thompson::State::Dense { .. }
| thompson::State::Fail
| thompson::State::Match { .. } => break,
thompson::State::Look { look, next } => {
if !look_have.contains(look) {
break;
}
id = next;
}
thompson::State::Union { ref alternates } => {
id = match alternates.get(0) {
None => break,
Some(&id) => id,
};
// We need to process our alternates in order to preserve
// match preferences, so put the earliest alternates closer
// to the top of the stack.
stack.extend(alternates[1..].iter().rev());
}
thompson::State::BinaryUnion { alt1, alt2 } => {
id = alt1;
stack.push(alt2);
}
thompson::State::Capture { next, .. } => {
id = next;
}
}
}
}
}
/// Add the NFA state IDs in the given `set` to the given DFA builder state.
/// The order in which states are added corresponds to the order in which they
/// were added to `set`.
///
/// The DFA builder state given should already have its complete set of match
/// pattern IDs added (if any) and any look-behind assertions (StartLF, Start
/// and whether this state is being generated for a transition over a word byte
/// when applicable) that are true immediately prior to transitioning into this
/// state (via `builder.look_have()`). The match pattern IDs should correspond
/// to matches that occurred on the previous transition, since all matches are
/// delayed by one byte. The things that should _not_ be set are look-ahead
/// assertions (EndLF, End and whether the next byte is a word byte or not).
/// The builder state should also not have anything in `look_need` set, as this
/// routine will compute that for you.
///
/// The given NFA should be able to resolve all identifiers in `set` to a
/// particular NFA state. Additionally, `set` must have capacity equivalent
/// to `nfa.len()`.
pub(crate) fn add_nfa_states(
nfa: &thompson::NFA,
set: &SparseSet,
builder: &mut StateBuilderNFA,
) {
for nfa_id in set.iter() {
match *nfa.state(nfa_id) {
thompson::State::ByteRange { .. } => {
builder.add_nfa_state_id(nfa_id);
}
thompson::State::Sparse { .. } => {
builder.add_nfa_state_id(nfa_id);
}
thompson::State::Dense { .. } => {
builder.add_nfa_state_id(nfa_id);
}
thompson::State::Look { look, .. } => {
builder.add_nfa_state_id(nfa_id);
builder.set_look_need(|need| need.insert(look));
}
thompson::State::Union { .. }
| thompson::State::BinaryUnion { .. } => {
// Pure epsilon transitions don't need to be tracked as part
// of the DFA state. Tracking them is actually superfluous;
// they won't cause any harm other than making determinization
// slower.
//
// Why aren't these needed? Well, in an NFA, epsilon
// transitions are really just jumping points to other states.
// So once you hit an epsilon transition, the same set of
// resulting states always appears. Therefore, putting them in
// a DFA's set of ordered NFA states is strictly redundant.
//
// Look-around states are also epsilon transitions, but
// they are *conditional*. So their presence could be
// discriminatory, and thus, they are tracked above.
//
// But wait... why are epsilon states in our `set` in the first
// place? Why not just leave them out? They're in our `set`
// because it was generated by computing an epsilon closure,
// and we want to keep track of all states we visited to avoid
// re-visiting them. In exchange, we have to do this second
// iteration over our collected states to finalize our DFA
// state. In theory, we could avoid this second iteration if
// we maintained two sets during epsilon closure: the set of
// visited states (to avoid cycles) and the set of states that
// will actually be used to construct the next DFA state.
//
// Note that this optimization requires that we re-compute the
// epsilon closure to account for look-ahead in 'next' *only
// when necessary*. Namely, only when the set of look-around
// assertions changes and only when those changes are within
// the set of assertions that are needed in order to step
// through the closure correctly. Otherwise, if we re-do the
// epsilon closure needlessly, it could change based on the
// fact that we are omitting epsilon states here.
//
// -----
//
// Welp, scratch the above. It turns out that recording these
// is in fact necessary to seemingly handle one particularly
// annoying case: when a conditional epsilon transition is
// put inside of a repetition operator. One specific case I
// ran into was the regex `(?:\b|%)+` on the haystack `z%`.
// The correct leftmost first matches are: [0, 0] and [1, 1].
// But the DFA was reporting [0, 0] and [1, 2]. To understand
// why this happens, consider the NFA for the aforementioned
// regex:
//
// >000000: binary-union(4, 1)
// 000001: \x00-\xFF => 0
// 000002: WordAscii => 5
// 000003: % => 5
// ^000004: binary-union(2, 3)
// 000005: binary-union(4, 6)
// 000006: MATCH(0)
//
// The problem here is that one of the DFA start states is
// going to consist of the NFA states [2, 3] by computing the
// epsilon closure of state 4. State 4 isn't included because
// we previously were not keeping track of union states. But
// only a subset of transitions out of this state will be able
// to follow WordAscii, and in those cases, the epsilon closure
// is redone. The only problem is that computing the epsilon
// closure from [2, 3] is different than computing the epsilon
// closure from [4]. In the former case, assuming the WordAscii
// assertion is satisfied, you get: [2, 3, 6]. In the latter
// case, you get: [2, 6, 3]. Notice that '6' is the match state
// and appears AFTER '3' in the former case. This leads to a
// preferential but incorrect match of '%' before returning
// a match. In the latter case, the match is preferred over
// continuing to accept the '%'.
//
// It almost feels like we might be able to fix the NFA states
// to avoid this, or to at least only keep track of union
// states where this actually matters, since in the vast
// majority of cases, this doesn't matter.
//
// Another alternative would be to define a new HIR property
// called "assertion is repeated anywhere" and compute it
// inductively over the entire pattern. If it happens anywhere,
// which is probably pretty rare, then we record union states.
// Otherwise we don't.
builder.add_nfa_state_id(nfa_id);
}
// Capture states we definitely do not need to record, since they
// are unconditional epsilon transitions with no branching.
thompson::State::Capture { .. } => {}
// It's not totally clear whether we need to record fail states or
// not, but we do so out of an abundance of caution. Since they are
// quite rare in practice, there isn't much cost to recording them.
thompson::State::Fail => {
builder.add_nfa_state_id(nfa_id);
}
thompson::State::Match { .. } => {
// Normally, the NFA match state doesn't actually need to
// be inside the DFA state. But since we delay matches by
// one byte, the matching DFA state corresponds to states
// that transition from the one we're building here. And
// the way we detect those cases is by looking for an NFA
// match state. See 'next' for how this is handled.
builder.add_nfa_state_id(nfa_id);
}
}
}
// If we know this state contains no look-around assertions, then
// there's no reason to track which look-around assertions were
// satisfied when this state was created.
if builder.look_need().is_empty() {
builder.set_look_have(|_| LookSet::empty());
}
}
/// Sets the appropriate look-behind assertions on the given state based on
/// this starting configuration.
pub(crate) fn set_lookbehind_from_start(
nfa: &thompson::NFA,
start: &Start,
builder: &mut StateBuilderMatches,
) {
let rev = nfa.is_reverse();
let lineterm = nfa.look_matcher().get_line_terminator();
let lookset = nfa.look_set_any();
match *start {
Start::NonWordByte => {
if lookset.contains_word() {
builder.set_look_have(|have| {
have.insert(Look::WordStartHalfAscii)
.insert(Look::WordStartHalfUnicode)
});
}
}
Start::WordByte => {
if lookset.contains_word() {
builder.set_is_from_word();
}
}
Start::Text => {
if lookset.contains_anchor_haystack() {
builder.set_look_have(|have| have.insert(Look::Start));
}
if lookset.contains_anchor_line() {
builder.set_look_have(|have| {
have.insert(Look::StartLF).insert(Look::StartCRLF)
});
}
if lookset.contains_word() {
builder.set_look_have(|have| {
have.insert(Look::WordStartHalfAscii)
.insert(Look::WordStartHalfUnicode)
});
}
}
Start::LineLF => {
if rev {
if lookset.contains_anchor_crlf() {
builder.set_is_half_crlf();
}
if lookset.contains_anchor_line() {
builder.set_look_have(|have| have.insert(Look::StartLF));
}
} else {
if lookset.contains_anchor_line() {
builder.set_look_have(|have| have.insert(Look::StartCRLF));
}
}
if lookset.contains_anchor_line() && lineterm == b'\n' {
builder.set_look_have(|have| have.insert(Look::StartLF));
}
if lookset.contains_word() {
builder.set_look_have(|have| {
have.insert(Look::WordStartHalfAscii)
.insert(Look::WordStartHalfUnicode)
});
}
}
Start::LineCR => {
if lookset.contains_anchor_crlf() {
if rev {
builder.set_look_have(|have| have.insert(Look::StartCRLF));
} else {
builder.set_is_half_crlf();
}
}
if lookset.contains_anchor_line() && lineterm == b'\r' {
builder.set_look_have(|have| have.insert(Look::StartLF));
}
if lookset.contains_word() {
builder.set_look_have(|have| {
have.insert(Look::WordStartHalfAscii)
.insert(Look::WordStartHalfUnicode)
});
}
}
Start::CustomLineTerminator => {
if lookset.contains_anchor_line() {
builder.set_look_have(|have| have.insert(Look::StartLF));
}
// This is a bit of a tricky case, but if the line terminator was
// set to a word byte, then we also need to behave as if the start
// configuration is Start::WordByte. That is, we need to mark our
// state as having come from a word byte.
if lookset.contains_word() {
if utf8::is_word_byte(lineterm) {
builder.set_is_from_word();
} else {
builder.set_look_have(|have| {
have.insert(Look::WordStartHalfAscii)
.insert(Look::WordStartHalfUnicode)
});
}
}
}
}
}
third_party/crates/regex-automata-0.4.18/src/util/determinize/state.rs
Vendored
+907
View File
@@ -0,0 +1,907 @@
/*!
This module defines a DFA state representation and builders for constructing
DFA states.
This representation is specifically for use in implementations of NFA-to-DFA
conversion via powerset construction. (Also called "determinization" in this
crate.)
The term "DFA state" is somewhat overloaded in this crate. In some cases, it
refers to the set of transitions over an alphabet for a particular state. In
other cases, it refers to a set of NFA states. The former is really about the
final representation of a state in a DFA's transition table, where as the
latter---what this module is focused on---is closer to an intermediate form
that is used to help eventually build the transition table.
This module exports four types. All four types represent the same idea: an
ordered set of NFA states. This ordered set represents the epsilon closure of a
particular NFA state, where the "epsilon closure" is the set of NFA states that
can be transitioned to without consuming any input. i.e., Follow all of the NFA
state's epsilon transitions. In addition, this implementation of DFA states
cares about two other things: the ordered set of pattern IDs corresponding
to the patterns that match if the state is a match state, and the set of
look-behind assertions that were true when the state was created.
The first, `State`, is a frozen representation of a state that cannot be
modified. It may be cheaply cloned without copying the state itself and can be
accessed safely from multiple threads simultaneously. This type is useful for
when one knows that the DFA state being constructed is distinct from any other
previously constructed states. Namely, powerset construction, in practice,
requires one to keep a cache of previously created DFA states. Otherwise,
the number of DFA states created in memory balloons to an impractically
large number. For this reason, equivalent states should endeavor to have an
equivalent byte-level representation. (In general, "equivalency" here means,
"equivalent assertions, pattern IDs and NFA state IDs." We do not require that
full DFA minimization be implemented here. This form of equivalency is only
surface deep and is more-or-less a practical necessity.)
The other three types represent different phases in the construction of a
DFA state. Internally, these three types (and `State`) all use the same
byte-oriented representation. That means one can use any of the builder types
to check whether the state it represents already exists or not. If it does,
then there is no need to freeze it into a `State` (which requires an alloc and
a copy). Here are the three types described succinctly:
* `StateBuilderEmpty` represents a state with no pattern IDs, no assertions
and no NFA states. Creating a `StateBuilderEmpty` performs no allocs. A
`StateBuilderEmpty` can only be used to query its underlying memory capacity,
or to convert into a builder for recording pattern IDs and/or assertions.
* `StateBuilderMatches` represents a state with zero or more pattern IDs, zero
or more satisfied assertions and zero NFA state IDs. A `StateBuilderMatches`
can only be used for adding pattern IDs and recording assertions.
* `StateBuilderNFA` represents a state with zero or more pattern IDs, zero or
more satisfied assertions and zero or more NFA state IDs. A `StateBuilderNFA`
can only be used for adding NFA state IDs and recording some assertions.
The expected flow here is to use the above builders to construct a candidate
DFA state to check if it already exists. If it does, then there's no need to
freeze it into a `State`. If it doesn't exist, then `StateBuilderNFA::to_state`
can be called to freeze the builder into an immutable `State`. In either
case, `clear` should be called on the builder to turn it back into a
`StateBuilderEmpty` that reuses the underlying memory.
The main purpose for splitting the builder into these distinct types is to
make it impossible to do things like adding a pattern ID after adding an NFA
state ID. Namely, this makes it simpler to use a space-and-time efficient
binary representation for the state. (The format is documented on the `Repr`
type below.) If we just used one type for everything, it would be possible for
callers to use an incorrect interleaving of calls and thus result in a corrupt
representation. I chose to use more type machinery to make this impossible to
do because 1) determinization is itself pretty complex and it wouldn't be too
hard to foul this up and 2) there isn't too much machinery involved and it's
well contained.
As an optimization, sometimes states won't have certain things set. For
example, if the underlying NFA has no word boundary assertions, then there is
no reason to set a state's look-behind assertion as to whether it was generated
from a word byte or not. Similarly, if a state has no NFA states corresponding
to look-around assertions, then there is no reason to set `look_have` to a
non-empty set. Finally, callers usually omit unconditional epsilon transitions
when adding NFA state IDs since they aren't discriminatory.
Finally, the binary representation used by these states is, thankfully, not
serialized anywhere. So any kind of change can be made with reckless abandon,
as long as everything in this module agrees.
*/
use core::mem;
use alloc::{sync::Arc, vec::Vec};
use crate::util::{
int::{I32, U32},
look::LookSet,
primitives::{PatternID, StateID},
wire::{self, Endian},
};
/// A DFA state that, at its core, is represented by an ordered set of NFA
/// states.
///
/// This type is intended to be used only in NFA-to-DFA conversion via powerset
/// construction.
///
/// It may be cheaply cloned and accessed safely from multiple threads
/// simultaneously.
#[derive(Clone, Eq, Hash, PartialEq, PartialOrd, Ord)]
pub(crate) struct State(Arc<[u8]>);
/// This Borrow impl permits us to lookup any state in a map by its byte
/// representation. This is particularly convenient when one has a StateBuilder
/// and we want to see if a correspondingly equivalent state already exists. If
/// one does exist, then we can reuse the allocation required by StateBuilder
/// without having to convert it into a State first.
impl core::borrow::Borrow<[u8]> for State {
fn borrow(&self) -> &[u8] {
&self.0
}
}
impl core::fmt::Debug for State {
fn fmt(&self, f: &mut core::fmt::Formatter) -> core::fmt::Result {
f.debug_tuple("State").field(&self.repr()).finish()
}
}
/// For docs on these routines, see the internal Repr and ReprVec types below.
impl State {
pub(crate) fn dead() -> State {
StateBuilderEmpty::new().into_matches().into_nfa().to_state()
}
pub(crate) fn is_match(&self) -> bool {
self.repr().is_match()
}
pub(crate) fn is_from_word(&self) -> bool {
self.repr().is_from_word()
}
pub(crate) fn is_half_crlf(&self) -> bool {
self.repr().is_half_crlf()
}
pub(crate) fn look_have(&self) -> LookSet {
self.repr().look_have()
}
pub(crate) fn look_need(&self) -> LookSet {
self.repr().look_need()
}
pub(crate) fn match_len(&self) -> usize {
self.repr().match_len()
}
pub(crate) fn match_pattern(&self, index: usize) -> PatternID {
self.repr().match_pattern(index)
}
pub(crate) fn match_pattern_ids(&self) -> Option<Vec<PatternID>> {
self.repr().match_pattern_ids()
}
#[cfg(all(test, not(miri)))]
pub(crate) fn iter_match_pattern_ids<F: FnMut(PatternID)>(&self, f: F) {
self.repr().iter_match_pattern_ids(f)
}
pub(crate) fn iter_nfa_state_ids<F: FnMut(StateID)>(&self, f: F) {
self.repr().iter_nfa_state_ids(f)
}
pub(crate) fn memory_usage(&self) -> usize {
self.0.len()
}
fn repr(&self) -> Repr<'_> {
Repr(&self.0)
}
}
/// A state builder that represents an empty state.
///
/// This is a useful "initial condition" for state construction. It has no
/// NFA state IDs, no assertions set and no pattern IDs. No allocations are
/// made when new() is called. Its main use is for being converted into a
/// builder that can capture assertions and pattern IDs.
#[derive(Clone, Debug)]
pub(crate) struct StateBuilderEmpty(Vec<u8>);
/// For docs on these routines, see the internal Repr and ReprVec types below.
impl StateBuilderEmpty {
pub(crate) fn new() -> StateBuilderEmpty {
StateBuilderEmpty(alloc::vec![])
}
pub(crate) fn into_matches(mut self) -> StateBuilderMatches {
self.0.extend_from_slice(&[0, 0, 0, 0, 0, 0, 0, 0, 0]);
StateBuilderMatches(self.0)
}
fn clear(&mut self) {
self.0.clear();
}
pub(crate) fn capacity(&self) -> usize {
self.0.capacity()
}
}
/// A state builder that collects assertions and pattern IDs.
///
/// When collecting pattern IDs is finished, this can be converted into a
/// builder that collects NFA state IDs.
#[derive(Clone)]
pub(crate) struct StateBuilderMatches(Vec<u8>);
impl core::fmt::Debug for StateBuilderMatches {
fn fmt(&self, f: &mut core::fmt::Formatter) -> core::fmt::Result {
f.debug_tuple("StateBuilderMatches").field(&self.repr()).finish()
}
}
/// For docs on these routines, see the internal Repr and ReprVec types below.
impl StateBuilderMatches {
pub(crate) fn into_nfa(mut self) -> StateBuilderNFA {
self.repr_vec().close_match_pattern_ids();
StateBuilderNFA { repr: self.0, prev_nfa_state_id: StateID::ZERO }
}
pub(crate) fn set_is_from_word(&mut self) {
self.repr_vec().set_is_from_word()
}
pub(crate) fn set_is_half_crlf(&mut self) {
self.repr_vec().set_is_half_crlf()
}
pub(crate) fn look_have(&self) -> LookSet {
LookSet::read_repr(&self.0[1..])
}
pub(crate) fn set_look_have(
&mut self,
set: impl FnMut(LookSet) -> LookSet,
) {
self.repr_vec().set_look_have(set)
}
pub(crate) fn add_match_pattern_id(&mut self, pid: PatternID) {
self.repr_vec().add_match_pattern_id(pid)
}
fn repr(&self) -> Repr<'_> {
Repr(&self.0)
}
fn repr_vec(&mut self) -> ReprVec<'_> {
ReprVec(&mut self.0)
}
}
/// A state builder that collects some assertions and NFA state IDs.
///
/// When collecting NFA state IDs is finished, this can be used to build a
/// `State` if necessary.
///
/// When dont with building a state (regardless of whether it got kept or not),
/// it's usually a good idea to call `clear` to get an empty builder back so
/// that it can be reused to build the next state.
#[derive(Clone)]
pub(crate) struct StateBuilderNFA {
repr: Vec<u8>,
prev_nfa_state_id: StateID,
}
impl core::fmt::Debug for StateBuilderNFA {
fn fmt(&self, f: &mut core::fmt::Formatter) -> core::fmt::Result {
f.debug_tuple("StateBuilderNFA").field(&self.repr()).finish()
}
}
/// For docs on these routines, see the internal Repr and ReprVec types below.
impl StateBuilderNFA {
pub(crate) fn to_state(&self) -> State {
State(Arc::from(&*self.repr))
}
pub(crate) fn clear(self) -> StateBuilderEmpty {
let mut builder = StateBuilderEmpty(self.repr);
builder.clear();
builder
}
pub(crate) fn look_need(&self) -> LookSet {
self.repr().look_need()
}
pub(crate) fn set_look_have(
&mut self,
set: impl FnMut(LookSet) -> LookSet,
) {
self.repr_vec().set_look_have(set)
}
pub(crate) fn set_look_need(
&mut self,
set: impl FnMut(LookSet) -> LookSet,
) {
self.repr_vec().set_look_need(set)
}
pub(crate) fn add_nfa_state_id(&mut self, sid: StateID) {
ReprVec(&mut self.repr)
.add_nfa_state_id(&mut self.prev_nfa_state_id, sid)
}
pub(crate) fn as_bytes(&self) -> &[u8] {
&self.repr
}
fn repr(&self) -> Repr<'_> {
Repr(&self.repr)
}
fn repr_vec(&mut self) -> ReprVec<'_> {
ReprVec(&mut self.repr)
}
}
/// Repr is a read-only view into the representation of a DFA state.
///
/// Primarily, a Repr is how we achieve DRY: we implement decoding the format
/// in one place, and then use a Repr to implement the various methods on the
/// public state types.
///
/// The format is as follows:
///
/// The first three bytes correspond to bitsets.
///
/// Byte 0 is a bitset corresponding to miscellaneous flags associated with the
/// state. Bit 0 is set to 1 if the state is a match state. Bit 1 is set to 1
/// if the state has pattern IDs explicitly written to it. (This is a flag that
/// is not meant to be set by determinization, but rather, is used as part of
/// an internal space-saving optimization.) Bit 2 is set to 1 if the state was
/// generated by a transition over a "word" byte. (Callers may not always set
/// this. For example, if the NFA has no word boundary assertion, then needing
/// to track whether a state came from a word byte or not is superfluous and
/// wasteful.) Bit 3 is set to 1 if the state was generated by a transition
/// from a `\r` (forward search) or a `\n` (reverse search) when CRLF mode is
/// enabled.
///
/// Bytes 1..5 correspond to the look-behind assertions that were satisfied
/// by the transition that created this state. (Look-ahead assertions are not
/// tracked as part of states. Instead, these are applied by re-computing the
/// epsilon closure of a state when computing the transition function. See
/// `next` in the parent module.)
///
/// Bytes 5..9 correspond to the set of look-around assertions (including both
/// look-behind and look-ahead) that appear somewhere in this state's set of
/// NFA state IDs. This is used to determine whether this state's epsilon
/// closure should be re-computed when computing the transition function.
/// Namely, look-around assertions are "just" conditional epsilon transitions,
/// so if there are new assertions available when computing the transition
/// function, we should only re-compute the epsilon closure if those new
/// assertions are relevant to this particular state.
///
/// Bytes 9..13 correspond to a 32-bit native-endian encoded integer
/// corresponding to the number of patterns encoded in this state. If the state
/// is not a match state (byte 0 bit 0 is 0) or if it's only pattern ID is
/// PatternID::ZERO, then no integer is encoded at this position. Instead, byte
/// offset 3 is the position at which the first NFA state ID is encoded.
///
/// For a match state with at least one non-ZERO pattern ID, the next bytes
/// correspond to a sequence of 32-bit native endian encoded integers that
/// represent each pattern ID, in order, that this match state represents.
///
/// After the pattern IDs (if any), NFA state IDs are delta encoded as
/// varints.[1] The first NFA state ID is encoded as itself, and each
/// subsequent NFA state ID is encoded as the difference between itself and the
/// previous NFA state ID.
///
/// [1] - https://developers.google.com/protocol-buffers/docs/encoding#varints
struct Repr<'a>(&'a [u8]);
impl<'a> Repr<'a> {
/// Returns true if and only if this is a match state.
///
/// If callers have added pattern IDs to this state, then callers MUST set
/// this state as a match state explicitly. However, as a special case,
/// states that are marked as match states but with no pattern IDs, then
/// the state is treated as if it had a single pattern ID equivalent to
/// PatternID::ZERO.
fn is_match(&self) -> bool {
self.0[0] & (1 << 0) > 0
}
/// Returns true if and only if this state has had at least one pattern
/// ID added to it.
///
/// This is an internal-only flag that permits the representation to save
/// space in the common case of an NFA with one pattern in it. In that
/// case, a match state can only ever have exactly one pattern ID:
/// PatternID::ZERO. So there's no need to represent it.
fn has_pattern_ids(&self) -> bool {
self.0[0] & (1 << 1) > 0
}
/// Returns true if and only if this state is marked as having been created
/// from a transition over a word byte. This is useful for checking whether
/// a word boundary assertion is true or not, which requires look-behind
/// (whether the current state came from a word byte or not) and look-ahead
/// (whether the transition byte is a word byte or not).
///
/// Since states with this set are distinct from states that don't have
/// this set (even if they are otherwise equivalent), callers should not
/// set this assertion unless the underlying NFA has at least one word
/// boundary assertion somewhere. Otherwise, a superfluous number of states
/// may be created.
fn is_from_word(&self) -> bool {
self.0[0] & (1 << 2) > 0
}
/// Returns true if and only if this state is marked as being inside of a
/// CRLF terminator. In the forward direction, this means the state was
/// created after seeing a `\r`. In the reverse direction, this means the
/// state was created after seeing a `\n`.
fn is_half_crlf(&self) -> bool {
self.0[0] & (1 << 3) > 0
}
/// The set of look-behind assertions that were true in the transition that
/// created this state.
///
/// Generally, this should be empty if 'look_need' is empty, since there is
/// no reason to track which look-behind assertions are true if the state
/// has no conditional epsilon transitions.
///
/// Satisfied look-ahead assertions are not tracked in states. Instead,
/// these are re-computed on demand via epsilon closure when computing the
/// transition function.
fn look_have(&self) -> LookSet {
LookSet::read_repr(&self.0[1..])
}
/// The set of look-around (both behind and ahead) assertions that appear
/// at least once in this state's set of NFA states.
///
/// This is used to determine whether the epsilon closure needs to be
/// re-computed when computing the transition function. Namely, if the
/// state has no conditional epsilon transitions, then there is no need
/// to re-compute the epsilon closure.
fn look_need(&self) -> LookSet {
LookSet::read_repr(&self.0[5..])
}
/// Returns the total number of match pattern IDs in this state.
///
/// If this state is not a match state, then this always returns 0.
fn match_len(&self) -> usize {
if !self.is_match() {
0
} else if !self.has_pattern_ids() {
1
} else {
self.encoded_pattern_len()
}
}
/// Returns the pattern ID for this match state at the given index.
///
/// If the given index is greater than or equal to `match_len()` for this
/// state, then this could panic or return incorrect results.
fn match_pattern(&self, index: usize) -> PatternID {
if !self.has_pattern_ids() {
PatternID::ZERO
} else {
let offset = 13 + index * PatternID::SIZE;
// This is OK since we only ever serialize valid PatternIDs to
// states.
wire::read_pattern_id_unchecked(&self.0[offset..]).0
}
}
/// Returns a copy of all match pattern IDs in this state. If this state
/// is not a match state, then this returns None.
fn match_pattern_ids(&self) -> Option<Vec<PatternID>> {
if !self.is_match() {
return None;
}
let mut pids = alloc::vec![];
self.iter_match_pattern_ids(|pid| pids.push(pid));
Some(pids)
}
/// Calls the given function on every pattern ID in this state.
fn iter_match_pattern_ids<F: FnMut(PatternID)>(&self, mut f: F) {
if !self.is_match() {
return;
}
// As an optimization for a very common case, when this is a match
// state for an NFA with only one pattern, we don't actually write the
// pattern ID to the state representation. Instead, we know it must
// be there since it is the only possible choice.
if !self.has_pattern_ids() {
f(PatternID::ZERO);
return;
}
let mut pids = &self.0[13..self.pattern_offset_end()];
while !pids.is_empty() {
let pid = wire::read_u32(pids);
pids = &pids[PatternID::SIZE..];
// This is OK since we only ever serialize valid PatternIDs to
// states. And since pattern IDs can never exceed a usize, the
// unwrap is OK.
f(PatternID::new_unchecked(usize::try_from(pid).unwrap()));
}
}
/// Calls the given function on every NFA state ID in this state.
fn iter_nfa_state_ids<F: FnMut(StateID)>(&self, mut f: F) {
let mut sids = &self.0[self.pattern_offset_end()..];
let mut prev = 0i32;
while !sids.is_empty() {
let (delta, nr) = read_vari32(sids);
sids = &sids[nr..];
let sid = prev + delta;
prev = sid;
// This is OK since we only ever serialize valid StateIDs to
// states. And since state IDs can never exceed an isize, they must
// always be able to fit into a usize, and thus cast is OK.
f(StateID::new_unchecked(sid.as_usize()))
}
}
/// Returns the offset into this state's representation where the pattern
/// IDs end and the NFA state IDs begin.
fn pattern_offset_end(&self) -> usize {
let encoded = self.encoded_pattern_len();
if encoded == 0 {
return 9;
}
// This arithmetic is OK since we were able to address this many bytes
// when writing to the state, thus, it must fit into a usize.
encoded.checked_mul(4).unwrap().checked_add(13).unwrap()
}
/// Returns the total number of *encoded* pattern IDs in this state.
///
/// This may return 0 even when this is a match state, since the pattern
/// ID `PatternID::ZERO` is not encoded when it's the only pattern ID in
/// the match state (the overwhelming common case).
fn encoded_pattern_len(&self) -> usize {
if !self.has_pattern_ids() {
return 0;
}
// This unwrap is OK since the total number of patterns is always
// guaranteed to fit into a usize.
usize::try_from(wire::read_u32(&self.0[9..13])).unwrap()
}
}
impl<'a> core::fmt::Debug for Repr<'a> {
fn fmt(&self, f: &mut core::fmt::Formatter) -> core::fmt::Result {
let mut nfa_ids = alloc::vec![];
self.iter_nfa_state_ids(|sid| nfa_ids.push(sid));
f.debug_struct("Repr")
.field("is_match", &self.is_match())
.field("is_from_word", &self.is_from_word())
.field("is_half_crlf", &self.is_half_crlf())
.field("look_have", &self.look_have())
.field("look_need", &self.look_need())
.field("match_pattern_ids", &self.match_pattern_ids())
.field("nfa_state_ids", &nfa_ids)
.finish()
}
}
/// ReprVec is a write-only view into the representation of a DFA state.
///
/// See Repr for more details on the purpose of this type and also the format.
///
/// Note that not all possible combinations of methods may be called. This is
/// precisely what the various StateBuilder types encapsulate: they only
/// permit valid combinations via Rust's linear typing.
struct ReprVec<'a>(&'a mut Vec<u8>);
impl<'a> ReprVec<'a> {
/// Set this state as a match state.
///
/// This should not be exposed explicitly outside of this module. It is
/// set automatically when a pattern ID is added.
fn set_is_match(&mut self) {
self.0[0] |= 1 << 0;
}
/// Set that this state has pattern IDs explicitly written to it.
///
/// This should not be exposed explicitly outside of this module. This is
/// used internally as a space saving optimization. Namely, if the state
/// is a match state but does not have any pattern IDs written to it,
/// then it is automatically inferred to have a pattern ID of ZERO.
fn set_has_pattern_ids(&mut self) {
self.0[0] |= 1 << 1;
}
/// Set this state as being built from a transition over a word byte.
///
/// Setting this is only necessary when one needs to deal with word
/// boundary assertions. Therefore, if the underlying NFA has no word
/// boundary assertions, callers should not set this.
fn set_is_from_word(&mut self) {
self.0[0] |= 1 << 2;
}
/// Set this state as having seen half of a CRLF terminator.
///
/// In the forward direction, this should be set when a `\r` has been seen.
/// In the reverse direction, this should be set when a `\n` has been seen.
fn set_is_half_crlf(&mut self) {
self.0[0] |= 1 << 3;
}
/// The set of look-behind assertions that were true in the transition that
/// created this state.
fn look_have(&self) -> LookSet {
self.repr().look_have()
}
/// The set of look-around (both behind and ahead) assertions that appear
/// at least once in this state's set of NFA states.
fn look_need(&self) -> LookSet {
self.repr().look_need()
}
/// Mutate the set of look-behind assertions that were true in the
/// transition that created this state.
fn set_look_have(&mut self, mut set: impl FnMut(LookSet) -> LookSet) {
set(self.look_have()).write_repr(&mut self.0[1..]);
}
/// Mutate the set of look-around (both behind and ahead) assertions that
/// appear at least once in this state's set of NFA states.
fn set_look_need(&mut self, mut set: impl FnMut(LookSet) -> LookSet) {
set(self.look_need()).write_repr(&mut self.0[5..]);
}
/// Add a pattern ID to this state. All match states must have at least
/// one pattern ID associated with it.
///
/// Callers must never add duplicative pattern IDs.
///
/// The order in which patterns are added must correspond to the order
/// in which patterns are reported as matches.
fn add_match_pattern_id(&mut self, pid: PatternID) {
// As a (somewhat small) space saving optimization, in the case where
// a matching state has exactly one pattern ID, PatternID::ZERO, we do
// not write either the pattern ID or the number of patterns encoded.
// Instead, all we do is set the 'is_match' bit on this state. Overall,
// this saves 8 bytes per match state for the overwhelming majority of
// match states.
//
// In order to know whether pattern IDs need to be explicitly read or
// not, we use another internal-only bit, 'has_pattern_ids', to
// indicate whether they have been explicitly written or not.
if !self.repr().has_pattern_ids() {
if pid == PatternID::ZERO {
self.set_is_match();
return;
}
// Make room for 'close_match_pattern_ids' to write the total
// number of pattern IDs written.
self.0.extend(core::iter::repeat(0).take(PatternID::SIZE));
self.set_has_pattern_ids();
// If this was already a match state, then the only way that's
// possible when the state doesn't have pattern IDs is if
// PatternID::ZERO was added by the caller previously. In this
// case, we are now adding a non-ZERO pattern ID after it, in
// which case, we want to make sure to represent ZERO explicitly
// now.
if self.repr().is_match() {
write_u32(self.0, 0)
} else {
// Otherwise, just make sure the 'is_match' bit is set.
self.set_is_match();
}
}
write_u32(self.0, pid.as_u32());
}
/// Indicate that no more pattern IDs will be added to this state.
///
/// Once this is called, callers must not call it or 'add_match_pattern_id'
/// again.
///
/// This should not be exposed explicitly outside of this module. It
/// should be called only when converting a StateBuilderMatches into a
/// StateBuilderNFA.
fn close_match_pattern_ids(&mut self) {
// If we never wrote any pattern IDs, then there's nothing to do here.
if !self.repr().has_pattern_ids() {
return;
}
let patsize = PatternID::SIZE;
let pattern_bytes = self.0.len() - 13;
// Every pattern ID uses 4 bytes, so number of bytes should be
// divisible by 4.
assert_eq!(pattern_bytes % patsize, 0);
// This unwrap is OK since we are guaranteed that the maximum number
// of possible patterns fits into a u32.
let count32 = u32::try_from(pattern_bytes / patsize).unwrap();
wire::NE::write_u32(count32, &mut self.0[9..13]);
}
/// Add an NFA state ID to this state. The order in which NFA states are
/// added matters. It is the caller's responsibility to ensure that
/// duplicate NFA state IDs are not added.
fn add_nfa_state_id(&mut self, prev: &mut StateID, sid: StateID) {
let delta = sid.as_i32() - prev.as_i32();
write_vari32(self.0, delta);
*prev = sid;
}
/// Return a read-only view of this state's representation.
fn repr(&self) -> Repr<'_> {
Repr(self.0.as_slice())
}
}
/// Write a signed 32-bit integer using zig-zag encoding.
///
/// https://developers.google.com/protocol-buffers/docs/encoding#varints
fn write_vari32(data: &mut Vec<u8>, n: i32) {
let mut un = n.to_bits() << 1;
if n < 0 {
un = !un;
}
write_varu32(data, un)
}
/// Read a signed 32-bit integer using zig-zag encoding. Also, return the
/// number of bytes read.
///
/// https://developers.google.com/protocol-buffers/docs/encoding#varints
fn read_vari32(data: &[u8]) -> (i32, usize) {
let (un, i) = read_varu32(data);
let mut n = i32::from_bits(un >> 1);
if un & 1 != 0 {
n = !n;
}
(n, i)
}
/// Write an unsigned 32-bit integer as a varint. In essence, `n` is written
/// as a sequence of bytes where all bytes except for the last one have the
/// most significant bit set. The least significant 7 bits correspond to the
/// actual bits of `n`. So in the worst case, a varint uses 5 bytes, but in
/// very common cases, it uses fewer than 4.
///
/// https://developers.google.com/protocol-buffers/docs/encoding#varints
fn write_varu32(data: &mut Vec<u8>, mut n: u32) {
while n >= 0b1000_0000 {
data.push(n.low_u8() | 0b1000_0000);
n >>= 7;
}
data.push(n.low_u8());
}
/// Read an unsigned 32-bit varint. Also, return the number of bytes read.
///
/// https://developers.google.com/protocol-buffers/docs/encoding#varints
fn read_varu32(data: &[u8]) -> (u32, usize) {
// N.B. We can assume correctness here since we know that all var-u32 are
// written with write_varu32. Hence, the 'as' uses and unchecked arithmetic
// is all okay.
let mut n: u32 = 0;
let mut shift: u32 = 0;
for (i, &b) in data.iter().enumerate() {
if b < 0b1000_0000 {
return (n | (u32::from(b) << shift), i + 1);
}
n |= (u32::from(b) & 0b0111_1111) << shift;
shift += 7;
}
(0, 0)
}
/// Push a native-endian encoded `n` on to `dst`.
fn write_u32(dst: &mut Vec<u8>, n: u32) {
use crate::util::wire::NE;
let start = dst.len();
dst.extend(core::iter::repeat(0).take(mem::size_of::<u32>()));
NE::write_u32(n, &mut dst[start..]);
}
#[cfg(test)]
mod tests {
use alloc::vec;
use quickcheck::quickcheck;
use super::*;
#[cfg(not(miri))]
quickcheck! {
fn prop_state_read_write_nfa_state_ids(sids: Vec<StateID>) -> bool {
// Builders states do not permit duplicate IDs.
let sids = dedup_state_ids(sids);
let mut b = StateBuilderEmpty::new().into_matches().into_nfa();
for &sid in &sids {
b.add_nfa_state_id(sid);
}
let s = b.to_state();
let mut got = vec![];
s.iter_nfa_state_ids(|sid| got.push(sid));
got == sids
}
fn prop_state_read_write_pattern_ids(pids: Vec<PatternID>) -> bool {
// Builders states do not permit duplicate IDs.
let pids = dedup_pattern_ids(pids);
let mut b = StateBuilderEmpty::new().into_matches();
for &pid in &pids {
b.add_match_pattern_id(pid);
}
let s = b.into_nfa().to_state();
let mut got = vec![];
s.iter_match_pattern_ids(|pid| got.push(pid));
got == pids
}
fn prop_state_read_write_nfa_state_and_pattern_ids(
sids: Vec<StateID>,
pids: Vec<PatternID>
) -> bool {
// Builders states do not permit duplicate IDs.
let sids = dedup_state_ids(sids);
let pids = dedup_pattern_ids(pids);
let mut b = StateBuilderEmpty::new().into_matches();
for &pid in &pids {
b.add_match_pattern_id(pid);
}
let mut b = b.into_nfa();
for &sid in &sids {
b.add_nfa_state_id(sid);
}
let s = b.to_state();
let mut got_pids = vec![];
s.iter_match_pattern_ids(|pid| got_pids.push(pid));
let mut got_sids = vec![];
s.iter_nfa_state_ids(|sid| got_sids.push(sid));
got_pids == pids && got_sids == sids
}
}
quickcheck! {
fn prop_read_write_varu32(n: u32) -> bool {
let mut buf = vec![];
write_varu32(&mut buf, n);
let (got, nread) = read_varu32(&buf);
nread == buf.len() && got == n
}
fn prop_read_write_vari32(n: i32) -> bool {
let mut buf = vec![];
write_vari32(&mut buf, n);
let (got, nread) = read_vari32(&buf);
nread == buf.len() && got == n
}
}
#[cfg(not(miri))]
fn dedup_state_ids(sids: Vec<StateID>) -> Vec<StateID> {
let mut set = alloc::collections::BTreeSet::new();
let mut deduped = vec![];
for sid in sids {
if set.contains(&sid) {
continue;
}
set.insert(sid);
deduped.push(sid);
}
deduped
}
#[cfg(not(miri))]
fn dedup_pattern_ids(pids: Vec<PatternID>) -> Vec<PatternID> {
let mut set = alloc::collections::BTreeSet::new();
let mut deduped = vec![];
for pid in pids {
if set.contains(&pid) {
continue;
}
set.insert(pid);
deduped.push(pid);
}
deduped
}
}
third_party/crates/regex-automata-0.4.18/src/util/empty.rs
Vendored
+265
View File
@@ -0,0 +1,265 @@
/*!
This module provides helper routines for dealing with zero-width matches.
The main problem being solved here is this:
1. The caller wants to search something that they know is valid UTF-8, such
as a Rust `&str`.
2. The regex used by the caller can match the empty string. For example, `a*`.
3. The caller should never get match offsets returned that occur within the
encoding of a UTF-8 codepoint. It is logically incorrect, and also means that,
e.g., slicing the `&str` at those offsets will lead to a panic.
So the question here is, how do we prevent the caller from getting match
offsets that split a codepoint? For example, strictly speaking, the regex `a*`
matches `☃` at the positions `[0, 0]`, `[1, 1]`, `[2, 2]` and `[3, 3]` since
the UTF-8 encoding of `☃` is `\xE2\x98\x83`. In particular, the `NFA` that
underlies all of the matching engines in this crate doesn't have anything in
its state graph that prevents matching between UTF-8 code units. Indeed, any
engine derived from the `NFA` will match at those positions by virtue of the
fact that the `NFA` is byte oriented. That is, its transitions are defined over
bytes and the matching engines work by proceeding one byte at a time.
(An alternative architecture would be to define the transitions in an `NFA`
over codepoints, or `char`. And then make the matching engines proceed by
decoding one codepoint at a time. This is a viable strategy, but it doesn't
work for DFA matching engines because designing a fast and memory efficient
transition table for an alphabet as large as Unicode is quite difficult. More
to the point, the top-level `regex` crate supports matching on arbitrary bytes
when Unicode mode is disabled and one is searching a `&[u8]`. So in that case,
you can't just limit yourself to decoding codepoints and matching those. You
really do need to be able to follow byte oriented transitions on the `NFA`.)
In an older version of the regex crate, we handled this case not in the regex
engine, but in the iterators over matches. Namely, since this case only arises
when the match is empty, we "just" incremented the next starting position
of the search by `N`, where `N` is the length of the codepoint encoded at
the current position. The alternative or more "natural" solution of just
incrementing by `1` would result in executing a search of `a*` on `☃` like
this:
* Start search at `0`.
* Found match at `[0, 0]`.
* Next start position is `0`.
* To avoid an infinite loop, since it's an empty match, increment by `1`.
* Start search at `1`.
* Found match at `[1, 1]`. Oops.
But if we instead incremented by `3` (the length in bytes of `☃`), then we get
the following:
* Start search at `0`.
* Found match at `[0, 0]`.
* Next start position is `0`.
* To avoid an infinite loop, since it's an empty match, increment by `3`.
* Start search at `3`.
* Found match at `[3, 3]`.
And we get the correct result. But does this technique work in all cases?
Crucially, it requires that a zero-width match that splits a codepoint never
occurs beyond the starting position of the search. Because if it did, merely
incrementing the start position by the number of bytes in the codepoint at
the current position wouldn't be enough. A zero-width match could just occur
anywhere. It turns out that it is _almost_ true. We can convince ourselves by
looking at all possible patterns that can match the empty string:
* Patterns like `a*`, `a{0}`, `(?:)`, `a|` and `|a` all unconditionally match
the empty string. That is, assuming there isn't an `a` at the current position,
they will all match the empty string at the start of a search. There is no way
to move past it because any other match would not be "leftmost."
* `^` only matches at the beginning of the haystack, where the start position
is `0`. Since we know we're searching valid UTF-8 (if it isn't valid UTF-8,
then this entire problem goes away because it implies your string type supports
invalid UTF-8 and thus must deal with offsets that not only split a codepoint
but occur in entirely invalid UTF-8 somehow), it follows that `^` never matches
between the code units of a codepoint because the start of a valid UTF-8 string
is never within the encoding of a codepoint.
* `$` basically the same logic as `^`, but for the end of a string. A valid
UTF-8 string can't have an incomplete codepoint at the end of it.
* `(?m:^)` follows similarly to `^`, but it can match immediately following
a `\n`. However, since a `\n` is always a codepoint itself and can never
appear within a codepoint, it follows that the position immediately following
a `\n` in a string that is valid UTF-8 is guaranteed to not be between the
code units of another codepoint. (One caveat here is that the line terminator
for multi-line anchors can now be changed to any arbitrary byte, including
things like `\x98` which might occur within a codepoint. However, this wasn't
supported by the old regex crate. If it was, it pose the same problems as
`(?-u:\B)`, as we'll discuss below.)
* `(?m:$)` a similar argument as for `(?m:^)`. The only difference is that a
`(?m:$)` matches just before a `\n`. But the same argument applies.
* `(?Rm:^)` and `(?Rm:$)` weren't supported by the old regex crate, but the
CRLF aware line anchors follow a similar argument as for `(?m:^)` and `(?m:$)`.
Namely, since they only ever match at a boundary where one side is either a
`\r` or a `\n`, neither of which can occur within a codepoint.
* `\b` only matches at positions where both sides are valid codepoints, so
this cannot split a codepoint.
* `\B`, like `\b`, also only matches at positions where both sides are valid
codepoints. So this cannot split a codepoint either.
* `(?-u:\b)` matches only at positions where at least one side of it is an ASCII
word byte. Since ASCII bytes cannot appear as code units in non-ASCII codepoints
(one of the many amazing qualities of UTF-8), it follows that this too cannot
split a codepoint.
* `(?-u:\B)` finally represents a problem. It can matches between *any* two
bytes that are either both word bytes or non-word bytes. Since code units like
`\xE2` and `\x98` (from the UTF-8 encoding of `☃`) are both non-word bytes,
`(?-u:\B)` will match at the position between them.
Thus, our approach of incrementing one codepoint at a time after seeing an
empty match is flawed because `(?-u:\B)` can result in an empty match that
splits a codepoint at a position past the starting point of a search. For
example, searching `(?-u:\B)` on `a☃` would produce the following matches: `[2,
2]`, `[3, 3]` and `[4, 4]`. The positions at `0` and `1` don't match because
they correspond to word boundaries since `a` is an ASCII word byte.
So what did the old regex crate do to avoid this? It banned `(?-u:\B)` from
regexes that could match `&str`. That might sound extreme, but a lot of other
things were banned too. For example, all of `(?-u:.)`, `(?-u:[^a])` and
`(?-u:\W)` can match invalid UTF-8 too, including individual code units with a
codepoint. The key difference is that those expressions could never produce an
empty match. That ban happens when translating an `Ast` to an `Hir`, because
that process that reason about whether an `Hir` can produce *non-empty* matches
at invalid UTF-8 boundaries. Bottom line though is that we side-stepped the
`(?-u:\B)` issue by banning it.
If banning `(?-u:\B)` were the only issue with the old regex crate's approach,
then I probably would have kept it. `\B` is rarely used, so it's not such a big
deal to have to work-around it. However, the problem with the above approach
is that it doesn't compose. The logic for avoiding splitting a codepoint only
lived in the iterator, which means if anyone wants to implement their own
iterator over regex matches, they have to deal with this extremely subtle edge
case to get full correctness.
Instead, in this crate, we take the approach of pushing this complexity down
to the lowest layers of each regex engine. The approach is pretty simple:
* If this corner case doesn't apply, don't do anything. (For example, if UTF-8
mode isn't enabled or if the regex cannot match the empty string.)
* If an empty match is reported, explicitly check if it splits a codepoint.
* If it doesn't, we're done, return the match.
* If it does, then ignore the match and re-run the search.
* Repeat the above process until the end of the haystack is reached or a match
is found that doesn't split a codepoint or isn't zero width.
And that's pretty much what this module provides. Every regex engine uses these
methods in their lowest level public APIs, but just above the layer where
their internal engine is used. That way, all regex engines can be arbitrarily
composed without worrying about handling this case, and iterators don't need to
handle it explicitly.
(It turns out that a new feature I added, support for changing the line
terminator in a regex to any arbitrary byte, also provokes the above problem.
Namely, the byte could be invalid UTF-8 or a UTF-8 continuation byte. So that
support would need to be limited or banned when UTF-8 mode is enabled, just
like we did for `(?-u:\B)`. But thankfully our more robust approach in this
crate handles that case just fine too.)
*/
use crate::util::search::{Input, MatchError};
#[cold]
#[inline(never)]
pub(crate) fn skip_splits_fwd<T, F>(
input: &Input<'_>,
init_value: T,
match_offset: usize,
find: F,
) -> Result<Option<T>, MatchError>
where
F: FnMut(&Input<'_>) -> Result<Option<(T, usize)>, MatchError>,
{
skip_splits(true, input, init_value, match_offset, find)
}
#[cold]
#[inline(never)]
pub(crate) fn skip_splits_rev<T, F>(
input: &Input<'_>,
init_value: T,
match_offset: usize,
find: F,
) -> Result<Option<T>, MatchError>
where
F: FnMut(&Input<'_>) -> Result<Option<(T, usize)>, MatchError>,
{
skip_splits(false, input, init_value, match_offset, find)
}
fn skip_splits<T, F>(
forward: bool,
input: &Input<'_>,
init_value: T,
mut match_offset: usize,
mut find: F,
) -> Result<Option<T>, MatchError>
where
F: FnMut(&Input<'_>) -> Result<Option<(T, usize)>, MatchError>,
{
// If our config says to do an anchored search, then we're definitely
// done. We just need to determine whether we have a valid match or
// not. If we don't, then we're not allowed to continue, so we report
// no match.
//
// This is actually quite a subtle correctness thing. The key here is
// that if we got an empty match that splits a codepoint after doing an
// anchored search in UTF-8 mode, then that implies that we must have
// *started* the search at a location that splits a codepoint. This
// follows from the fact that if a match is reported from an anchored
// search, then the start offset of the match *must* match the start
// offset of the search.
//
// It also follows that no other non-empty match is possible. For
// example, you might write a regex like '(?:)|SOMETHING' and start its
// search in the middle of a codepoint. The first branch is an empty
// regex that will bubble up a match at the first position, and then
// get rejected here and report no match. But what if 'SOMETHING' could
// have matched? We reason that such a thing is impossible, because
// if it does, it must report a match that starts in the middle of a
// codepoint. This in turn implies that a match is reported whose span
// does not correspond to valid UTF-8, and this breaks the promise
// made when UTF-8 mode is enabled. (That promise *can* be broken, for
// example, by enabling UTF-8 mode but building an by hand NFA that
// produces non-empty matches that span invalid UTF-8. This is an unchecked
// but documented precondition violation of UTF-8 mode, and is documented
// to have unspecified behavior.)
//
// I believe this actually means that if an anchored search is run, and
// UTF-8 mode is enabled and the start position splits a codepoint,
// then it is correct to immediately report no match without even
// executing the regex engine. But it doesn't really seem worth writing
// out that case in every regex engine to save a tiny bit of work in an
// extremely pathological case, so we just handle it here.
if input.get_anchored().is_anchored() {
return Ok(if input.is_char_boundary(match_offset) {
Some(init_value)
} else {
None
});
}
// Otherwise, we have an unanchored search, so just keep looking for
// matches until we have one that does not split a codepoint or we hit
// EOI.
let mut value = init_value;
let mut input = input.clone();
while !input.is_char_boundary(match_offset) {
if forward {
// The unwrap is OK here because overflowing usize while
// iterating over a slice is impossible, at it would require
// a slice of length greater than isize::MAX, which is itself
// impossible.
input.set_start(input.start().checked_add(1).unwrap());
} else {
input.set_end(match input.end().checked_sub(1) {
None => return Ok(None),
Some(end) => end,
});
}
match find(&input)? {
None => return Ok(None),
Some((new_value, new_match_end)) => {
value = new_value;
match_offset = new_match_end;
}
}
}
Ok(Some(value))
}
third_party/crates/regex-automata-0.4.18/src/util/escape.rs
Vendored
+84
View File
@@ -0,0 +1,84 @@
/*!
Provides convenience routines for escaping raw bytes.
Since this crate tends to deal with `&[u8]` everywhere and the default
`Debug` implementation just shows decimal integers, it makes debugging those
representations quite difficult. This module provides types that show `&[u8]`
as if it were a string, with invalid UTF-8 escaped into its byte-by-byte hex
representation.
*/
use crate::util::utf8;
/// Provides a convenient `Debug` implementation for a `u8`.
///
/// The `Debug` impl treats the byte as an ASCII, and emits a human readable
/// representation of it. If the byte isn't ASCII, then it's emitted as a hex
/// escape sequence.
#[derive(Clone, Copy)]
pub struct DebugByte(pub u8);
impl core::fmt::Debug for DebugByte {
fn fmt(&self, f: &mut core::fmt::Formatter) -> core::fmt::Result {
// Special case ASCII space. It's too hard to read otherwise, so
// put quotes around it. I sometimes wonder whether just '\x20' would
// be better...
if self.0 == b' ' {
return write!(f, "' '");
}
// 10 bytes is enough to cover any output from ascii::escape_default.
let mut bytes = [0u8; 10];
let mut len = 0;
for (i, mut b) in core::ascii::escape_default(self.0).enumerate() {
// capitalize \xab to \xAB
if i >= 2 && b'a' <= b && b <= b'f' {
b -= 32;
}
bytes[len] = b;
len += 1;
}
write!(f, "{}", core::str::from_utf8(&bytes[..len]).unwrap())
}
}
/// Provides a convenient `Debug` implementation for `&[u8]`.
///
/// This generally works best when the bytes are presumed to be mostly UTF-8,
/// but will work for anything. For any bytes that aren't UTF-8, they are
/// emitted as hex escape sequences.
pub struct DebugHaystack<'a>(pub &'a [u8]);
impl<'a> core::fmt::Debug for DebugHaystack<'a> {
fn fmt(&self, f: &mut core::fmt::Formatter) -> core::fmt::Result {
write!(f, "\"")?;
// This is a sad re-implementation of a similar impl found in bstr.
let mut bytes = self.0;
while let Some(result) = utf8::decode(bytes) {
let ch = match result {
Ok(ch) => ch,
Err(byte) => {
write!(f, r"\x{byte:02x}")?;
bytes = &bytes[1..];
continue;
}
};
bytes = &bytes[ch.len_utf8()..];
match ch {
'\0' => write!(f, "\\0")?,
// ASCII control characters except \0, \n, \r, \t
'\x01'..='\x08'
| '\x0b'
| '\x0c'
| '\x0e'..='\x19'
| '\x7f' => {
write!(f, "\\x{:02x}", u32::from(ch))?;
}
'\n' | '\r' | '\t' | _ => {
write!(f, "{}", ch.escape_debug())?;
}
}
}
write!(f, "\"")?;
Ok(())
}
}
third_party/crates/regex-automata-0.4.18/src/util/int.rs
Vendored
+246
View File
@@ -0,0 +1,246 @@
/*!
This module provides several integer oriented traits for converting between
both fixed size integers and integers whose size varies based on the target
(like `usize`).
The driving design principle of this module is to attempt to centralize as many
`as` casts as possible here. And in particular, we separate casts into two
buckets:
* Casts that we use for their truncating behavior. In this case, we use more
descriptive names, like `low_u32` and `high_u32`.
* Casts that we use for converting back-and-forth between `usize`. These
conversions are generally necessary because we often store indices in different
formats to save on memory, which requires converting to and from `usize`. In
this case, we very specifically do not want to overflow, and so the methods
defined here will panic if the `as` cast would be lossy in debug mode. (A
normal `as` cast will never panic!)
For `as` casts between raw pointers, we use `cast`, so `as` isn't needed there.
For regex engines, floating point is just never used, so we don't have to worry
about `as` casts for those.
Otherwise, this module pretty much covers all of our `as` needs except for one
thing: const contexts. There are a select few places in this crate where we
still need to use `as` because const functions on traits aren't stable yet.
If we wind up significantly expanding our const footprint in this crate, it
might be worth defining free functions to handle those cases. But at the time
of writing, that just seemed like too much ceremony. Instead, I comment each
such use of `as` in a const context with a "fixme" notice.
NOTE: for simplicity, we don't take target pointer width into account here for
`usize` conversions. Since we currently only panic in debug mode, skipping the
check when it can be proven it isn't needed at compile time doesn't really
matter. Now, if we wind up wanting to do as many checks as possible in release
mode, then we would want to skip those when we know the conversions are always
non-lossy.
NOTE: this module isn't an exhaustive API. For example, we still use things
like `u64::from` where possible, or even `usize::try_from()` for when we do
explicitly want to panic or when we want to return an error for overflow.
*/
// We define a little more than what we need, but I'd rather just have
// everything via a consistent and uniform API then have holes.
#![allow(dead_code)]
pub(crate) trait U8 {
fn as_usize(self) -> usize;
}
impl U8 for u8 {
fn as_usize(self) -> usize {
usize::from(self)
}
}
pub(crate) trait U16 {
fn as_usize(self) -> usize;
fn low_u8(self) -> u8;
fn high_u8(self) -> u8;
}
impl U16 for u16 {
fn as_usize(self) -> usize {
usize::from(self)
}
fn low_u8(self) -> u8 {
self as u8
}
fn high_u8(self) -> u8 {
(self >> 8) as u8
}
}
pub(crate) trait U32 {
fn as_usize(self) -> usize;
fn low_u8(self) -> u8;
fn low_u16(self) -> u16;
fn high_u16(self) -> u16;
}
impl U32 for u32 {
fn as_usize(self) -> usize {
#[cfg(debug_assertions)]
{
usize::try_from(self).expect("u32 overflowed usize")
}
#[cfg(not(debug_assertions))]
{
self as usize
}
}
fn low_u8(self) -> u8 {
self as u8
}
fn low_u16(self) -> u16 {
self as u16
}
fn high_u16(self) -> u16 {
(self >> 16) as u16
}
}
pub(crate) trait U64 {
fn as_usize(self) -> usize;
fn low_u8(self) -> u8;
fn low_u16(self) -> u16;
fn low_u32(self) -> u32;
fn high_u32(self) -> u32;
}
impl U64 for u64 {
fn as_usize(self) -> usize {
#[cfg(debug_assertions)]
{
usize::try_from(self).expect("u64 overflowed usize")
}
#[cfg(not(debug_assertions))]
{
self as usize
}
}
fn low_u8(self) -> u8 {
self as u8
}
fn low_u16(self) -> u16 {
self as u16
}
fn low_u32(self) -> u32 {
self as u32
}
fn high_u32(self) -> u32 {
(self >> 32) as u32
}
}
pub(crate) trait I32 {
fn as_usize(self) -> usize;
fn to_bits(self) -> u32;
fn from_bits(n: u32) -> i32;
}
impl I32 for i32 {
fn as_usize(self) -> usize {
#[cfg(debug_assertions)]
{
usize::try_from(self).expect("i32 overflowed usize")
}
#[cfg(not(debug_assertions))]
{
self as usize
}
}
fn to_bits(self) -> u32 {
self as u32
}
fn from_bits(n: u32) -> i32 {
n as i32
}
}
pub(crate) trait Usize {
fn as_u8(self) -> u8;
fn as_u16(self) -> u16;
fn as_u32(self) -> u32;
fn as_u64(self) -> u64;
}
impl Usize for usize {
fn as_u8(self) -> u8 {
#[cfg(debug_assertions)]
{
u8::try_from(self).expect("usize overflowed u8")
}
#[cfg(not(debug_assertions))]
{
self as u8
}
}
fn as_u16(self) -> u16 {
#[cfg(debug_assertions)]
{
u16::try_from(self).expect("usize overflowed u16")
}
#[cfg(not(debug_assertions))]
{
self as u16
}
}
fn as_u32(self) -> u32 {
#[cfg(debug_assertions)]
{
u32::try_from(self).expect("usize overflowed u32")
}
#[cfg(not(debug_assertions))]
{
self as u32
}
}
fn as_u64(self) -> u64 {
#[cfg(debug_assertions)]
{
u64::try_from(self).expect("usize overflowed u64")
}
#[cfg(not(debug_assertions))]
{
self as u64
}
}
}
// Pointers aren't integers, but we convert pointers to integers to perform
// offset arithmetic in some places. (And no, we don't convert the integers
// back to pointers.) So add 'as_usize' conversions here too for completeness.
//
// These 'as' casts are actually okay because they're always non-lossy. But the
// idea here is to just try and remove as much 'as' as possible, particularly
// in this crate where we are being really paranoid about offsets and making
// sure we don't panic on inputs that might be untrusted. This way, the 'as'
// casts become easier to audit if they're all in one place, even when some of
// them are actually okay 100% of the time.
pub(crate) trait Pointer {
fn as_usize(self) -> usize;
}
impl<T> Pointer for *const T {
fn as_usize(self) -> usize {
self as usize
}
}
third_party/crates/regex-automata-0.4.18/src/util/interpolate.rs
Vendored
+576
View File
@@ -0,0 +1,576 @@
/*!
Provides routines for interpolating capture group references.
That is, if a replacement string contains references like `$foo` or `${foo1}`,
then they are replaced with the corresponding capture values for the groups
named `foo` and `foo1`, respectively. Similarly, syntax like `$1` and `${1}`
is supported as well, with `1` corresponding to a capture group index and not
a name.
This module provides the free functions [`string`] and [`bytes`], which
interpolate Rust Unicode strings and byte strings, respectively.
# Format
These routines support two different kinds of capture references: unbraced and
braced.
For the unbraced format, the format supported is `$ref` where `name` can be
any character in the class `[0-9A-Za-z_]`. `ref` is always the longest
possible parse. So for example, `$1a` corresponds to the capture group named
`1a` and not the capture group at index `1`. If `ref` matches `^[0-9]+$`, then
it is treated as a capture group index itself and not a name.
For the braced format, the format supported is `${ref}` where `ref` can be any
sequence of bytes except for `}`. If no closing brace occurs, then it is not
considered a capture reference. As with the unbraced format, if `ref` matches
`^[0-9]+$`, then it is treated as a capture group index and not a name.
The braced format is useful for exerting precise control over the name of the
capture reference. For example, `${1}a` corresponds to the capture group
reference `1` followed by the letter `a`, where as `$1a` (as mentioned above)
corresponds to the capture group reference `1a`. The braced format is also
useful for expressing capture group names that use characters not supported by
the unbraced format. For example, `${foo[bar].baz}` refers to the capture group
named `foo[bar].baz`.
If a capture group reference is found and it does not refer to a valid capture
group, then it will be replaced with the empty string.
To write a literal `$`, use `$$`.
To be clear, and as exhibited via the type signatures in the routines in this
module, it is impossible for a replacement string to be invalid. A replacement
string may not have the intended semantics, but the interpolation procedure
itself can never fail.
*/
use alloc::{string::String, vec::Vec};
use crate::util::memchr::memchr;
/// Accepts a replacement string and interpolates capture references with their
/// corresponding values.
///
/// `append` should be a function that appends the string value of a capture
/// group at a particular index to the string given. If the capture group
/// index is invalid, then nothing should be appended.
///
/// `name_to_index` should be a function that maps a capture group name to a
/// capture group index. If the given name doesn't exist, then `None` should
/// be returned.
///
/// Finally, `dst` is where the final interpolated contents should be written.
/// If `replacement` contains no capture group references, then `dst` will be
/// equivalent to `replacement`.
///
/// See the [module documentation](self) for details about the format
/// supported.
///
/// # Example
///
/// ```
/// use regex_automata::util::interpolate;
///
/// let mut dst = String::new();
/// interpolate::string(
/// "foo $bar baz",
/// |index, dst| {
/// if index == 0 {
/// dst.push_str("BAR");
/// }
/// },
/// |name| {
/// if name == "bar" {
/// Some(0)
/// } else {
/// None
/// }
/// },
/// &mut dst,
/// );
/// assert_eq!("foo BAR baz", dst);
/// ```
pub fn string(
mut replacement: &str,
mut append: impl FnMut(usize, &mut String),
mut name_to_index: impl FnMut(&str) -> Option<usize>,
dst: &mut String,
) {
while !replacement.is_empty() {
match memchr(b'$', replacement.as_bytes()) {
None => break,
Some(i) => {
dst.push_str(&replacement[..i]);
replacement = &replacement[i..];
}
}
// Handle escaping of '$'.
if replacement.as_bytes().get(1).map_or(false, |&b| b == b'$') {
dst.push_str("$");
replacement = &replacement[2..];
continue;
}
debug_assert!(!replacement.is_empty());
let cap_ref = match find_cap_ref(replacement.as_bytes()) {
Some(cap_ref) => cap_ref,
None => {
dst.push_str("$");
replacement = &replacement[1..];
continue;
}
};
replacement = &replacement[cap_ref.end..];
match cap_ref.cap {
Ref::Number(i) => append(i, dst),
Ref::Named(name) => {
if let Some(i) = name_to_index(name) {
append(i, dst);
}
}
}
}
dst.push_str(replacement);
}
/// Accepts a replacement byte string and interpolates capture references with
/// their corresponding values.
///
/// `append` should be a function that appends the byte string value of a
/// capture group at a particular index to the byte string given. If the
/// capture group index is invalid, then nothing should be appended.
///
/// `name_to_index` should be a function that maps a capture group name to a
/// capture group index. If the given name doesn't exist, then `None` should
/// be returned.
///
/// Finally, `dst` is where the final interpolated contents should be written.
/// If `replacement` contains no capture group references, then `dst` will be
/// equivalent to `replacement`.
///
/// See the [module documentation](self) for details about the format
/// supported.
///
/// # Example
///
/// ```
/// use regex_automata::util::interpolate;
///
/// let mut dst = vec![];
/// interpolate::bytes(
/// b"foo $bar baz",
/// |index, dst| {
/// if index == 0 {
/// dst.extend_from_slice(b"BAR");
/// }
/// },
/// |name| {
/// if name == "bar" {
/// Some(0)
/// } else {
/// None
/// }
/// },
/// &mut dst,
/// );
/// assert_eq!(&b"foo BAR baz"[..], dst);
/// ```
pub fn bytes(
mut replacement: &[u8],
mut append: impl FnMut(usize, &mut Vec<u8>),
mut name_to_index: impl FnMut(&str) -> Option<usize>,
dst: &mut Vec<u8>,
) {
while !replacement.is_empty() {
match memchr(b'$', replacement) {
None => break,
Some(i) => {
dst.extend_from_slice(&replacement[..i]);
replacement = &replacement[i..];
}
}
// Handle escaping of '$'.
if replacement.get(1).map_or(false, |&b| b == b'$') {
dst.push(b'$');
replacement = &replacement[2..];
continue;
}
debug_assert!(!replacement.is_empty());
let cap_ref = match find_cap_ref(replacement) {
Some(cap_ref) => cap_ref,
None => {
dst.push(b'$');
replacement = &replacement[1..];
continue;
}
};
replacement = &replacement[cap_ref.end..];
match cap_ref.cap {
Ref::Number(i) => append(i, dst),
Ref::Named(name) => {
if let Some(i) = name_to_index(name) {
append(i, dst);
}
}
}
}
dst.extend_from_slice(replacement);
}
/// `CaptureRef` represents a reference to a capture group inside some text.
/// The reference is either a capture group name or a number.
///
/// It is also tagged with the position in the text following the
/// capture reference.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
struct CaptureRef<'a> {
cap: Ref<'a>,
end: usize,
}
/// A reference to a capture group in some text.
///
/// e.g., `$2`, `$foo`, `${foo}`.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum Ref<'a> {
Named(&'a str),
Number(usize),
}
impl<'a> From<&'a str> for Ref<'a> {
fn from(x: &'a str) -> Ref<'a> {
Ref::Named(x)
}
}
impl From<usize> for Ref<'static> {
fn from(x: usize) -> Ref<'static> {
Ref::Number(x)
}
}
/// Parses a possible reference to a capture group name in the given text,
/// starting at the beginning of `replacement`.
///
/// If no such valid reference could be found, None is returned.
///
/// Note that this returns a "possible" reference because this routine doesn't
/// know whether the reference is to a valid group or not. If it winds up not
/// being a valid reference, then it should be replaced with the empty string.
fn find_cap_ref(replacement: &[u8]) -> Option<CaptureRef<'_>> {
let mut i = 0;
let rep: &[u8] = replacement;
if rep.len() <= 1 || rep[0] != b'$' {
return None;
}
i += 1;
if rep[i] == b'{' {
return find_cap_ref_braced(rep, i + 1);
}
let mut cap_end = i;
while rep.get(cap_end).copied().map_or(false, is_valid_cap_letter) {
cap_end += 1;
}
if cap_end == i {
return None;
}
// We just verified that the range 0..cap_end is valid ASCII, so it must
// therefore be valid UTF-8. If we really cared, we could avoid this UTF-8
// check via an unchecked conversion or by parsing the number straight from
// &[u8].
let cap = core::str::from_utf8(&rep[i..cap_end])
.expect("valid UTF-8 capture name");
Some(CaptureRef {
cap: match cap.parse::<usize>() {
Ok(i) => Ref::Number(i),
Err(_) => Ref::Named(cap),
},
end: cap_end,
})
}
/// Looks for a braced reference, e.g., `${foo1}`. This assumes that an opening
/// brace has been found at `i-1` in `rep`. This then looks for a closing
/// brace and returns the capture reference within the brace.
fn find_cap_ref_braced(rep: &[u8], mut i: usize) -> Option<CaptureRef<'_>> {
assert_eq!(b'{', rep[i.checked_sub(1).unwrap()]);
let start = i;
while rep.get(i).map_or(false, |&b| b != b'}') {
i += 1;
}
if !rep.get(i).map_or(false, |&b| b == b'}') {
return None;
}
// When looking at braced names, we don't put any restrictions on the name,
// so it's possible it could be invalid UTF-8. But a capture group name
// can never be invalid UTF-8, so if we have invalid UTF-8, then we can
// safely return None.
let cap = match core::str::from_utf8(&rep[start..i]) {
Err(_) => return None,
Ok(cap) => cap,
};
Some(CaptureRef {
cap: match cap.parse::<usize>() {
Ok(i) => Ref::Number(i),
Err(_) => Ref::Named(cap),
},
end: i + 1,
})
}
/// Returns true if and only if the given byte is allowed in a capture name
/// written in non-brace form.
fn is_valid_cap_letter(b: u8) -> bool {
matches!(b, b'0'..=b'9' | b'a'..=b'z' | b'A'..=b'Z' | b'_')
}
#[cfg(test)]
mod tests {
use alloc::{string::String, vec, vec::Vec};
use super::{find_cap_ref, CaptureRef};
macro_rules! find {
($name:ident, $text:expr) => {
#[test]
fn $name() {
assert_eq!(None, find_cap_ref($text.as_bytes()));
}
};
($name:ident, $text:expr, $capref:expr) => {
#[test]
fn $name() {
assert_eq!(Some($capref), find_cap_ref($text.as_bytes()));
}
};
}
macro_rules! c {
($name_or_number:expr, $pos:expr) => {
CaptureRef { cap: $name_or_number.into(), end: $pos }
};
}
find!(find_cap_ref1, "$foo", c!("foo", 4));
find!(find_cap_ref2, "${foo}", c!("foo", 6));
find!(find_cap_ref3, "$0", c!(0, 2));
find!(find_cap_ref4, "$5", c!(5, 2));
find!(find_cap_ref5, "$10", c!(10, 3));
// See https://github.com/rust-lang/regex/pull/585
// for more on characters following numbers
find!(find_cap_ref6, "$42a", c!("42a", 4));
find!(find_cap_ref7, "${42}a", c!(42, 5));
find!(find_cap_ref8, "${42");
find!(find_cap_ref9, "${42 ");
find!(find_cap_ref10, " $0 ");
find!(find_cap_ref11, "$");
find!(find_cap_ref12, " ");
find!(find_cap_ref13, "");
find!(find_cap_ref14, "$1-$2", c!(1, 2));
find!(find_cap_ref15, "$1_$2", c!("1_", 3));
find!(find_cap_ref16, "$x-$y", c!("x", 2));
find!(find_cap_ref17, "$x_$y", c!("x_", 3));
find!(find_cap_ref18, "${#}", c!("#", 4));
find!(find_cap_ref19, "${Z[}", c!("Z[", 5));
find!(find_cap_ref20, "${¾}", c!("¾", 5));
find!(find_cap_ref21, "${¾a}", c!("¾a", 6));
find!(find_cap_ref22, "${a¾}", c!("a¾", 6));
find!(find_cap_ref23, "${☃}", c!("☃", 6));
find!(find_cap_ref24, "${a☃}", c!("a☃", 7));
find!(find_cap_ref25, "${☃a}", c!("☃a", 7));
find!(find_cap_ref26, "${名字}", c!("名字", 9));
fn interpolate_string(
mut name_to_index: Vec<(&'static str, usize)>,
caps: Vec<&'static str>,
replacement: &str,
) -> String {
name_to_index.sort_by_key(|x| x.0);
let mut dst = String::new();
super::string(
replacement,
|i, dst| {
if let Some(&s) = caps.get(i) {
dst.push_str(s);
}
},
|name| -> Option<usize> {
name_to_index
.binary_search_by_key(&name, |x| x.0)
.ok()
.map(|i| name_to_index[i].1)
},
&mut dst,
);
dst
}
fn interpolate_bytes(
mut name_to_index: Vec<(&'static str, usize)>,
caps: Vec<&'static str>,
replacement: &str,
) -> String {
name_to_index.sort_by_key(|x| x.0);
let mut dst = vec![];
super::bytes(
replacement.as_bytes(),
|i, dst| {
if let Some(&s) = caps.get(i) {
dst.extend_from_slice(s.as_bytes());
}
},
|name| -> Option<usize> {
name_to_index
.binary_search_by_key(&name, |x| x.0)
.ok()
.map(|i| name_to_index[i].1)
},
&mut dst,
);
String::from_utf8(dst).unwrap()
}
macro_rules! interp {
($name:ident, $map:expr, $caps:expr, $hay:expr, $expected:expr $(,)*) => {
#[test]
fn $name() {
assert_eq!(
$expected,
interpolate_string($map, $caps, $hay),
"interpolate::string failed",
);
assert_eq!(
$expected,
interpolate_bytes($map, $caps, $hay),
"interpolate::bytes failed",
);
}
};
}
interp!(
interp1,
vec![("foo", 2)],
vec!["", "", "xxx"],
"test $foo test",
"test xxx test",
);
interp!(
interp2,
vec![("foo", 2)],
vec!["", "", "xxx"],
"test$footest",
"test",
);
interp!(
interp3,
vec![("foo", 2)],
vec!["", "", "xxx"],
"test${foo}test",
"testxxxtest",
);
interp!(
interp4,
vec![("foo", 2)],
vec!["", "", "xxx"],
"test$2test",
"test",
);
interp!(
interp5,
vec![("foo", 2)],
vec!["", "", "xxx"],
"test${2}test",
"testxxxtest",
);
interp!(
interp6,
vec![("foo", 2)],
vec!["", "", "xxx"],
"test $$foo test",
"test $foo test",
);
interp!(
interp7,
vec![("foo", 2)],
vec!["", "", "xxx"],
"test $foo",
"test xxx",
);
interp!(
interp8,
vec![("foo", 2)],
vec!["", "", "xxx"],
"$foo test",
"xxx test",
);
interp!(
interp9,
vec![("bar", 1), ("foo", 2)],
vec!["", "yyy", "xxx"],
"test $bar$foo",
"test yyyxxx",
);
interp!(
interp10,
vec![("bar", 1), ("foo", 2)],
vec!["", "yyy", "xxx"],
"test $ test",
"test $ test",
);
interp!(
interp11,
vec![("bar", 1), ("foo", 2)],
vec!["", "yyy", "xxx"],
"test ${} test",
"test test",
);
interp!(
interp12,
vec![("bar", 1), ("foo", 2)],
vec!["", "yyy", "xxx"],
"test ${ } test",
"test test",
);
interp!(
interp13,
vec![("bar", 1), ("foo", 2)],
vec!["", "yyy", "xxx"],
"test ${a b} test",
"test test",
);
interp!(
interp14,
vec![("bar", 1), ("foo", 2)],
vec!["", "yyy", "xxx"],
"test ${a} test",
"test test",
);
// This is a funny case where a braced reference is never closed, but
// within the unclosed braced reference, there is an unbraced reference.
// In this case, the braced reference is just treated literally and the
// unbraced reference is found.
interp!(
interp15,
vec![("bar", 1), ("foo", 2)],
vec!["", "yyy", "xxx"],
"test ${wat $bar ok",
"test ${wat yyy ok",
);
}
third_party/crates/regex-automata-0.4.18/src/util/iter.rs
Vendored
+1022
View File
File diff suppressed because it is too large. Load diff
third_party/crates/regex-automata-0.4.18/src/util/lazy.rs
Vendored
+461
View File
@@ -0,0 +1,461 @@
/*!
A lazily initialized value for safe sharing between threads.
The principal type in this module is `Lazy`, which makes it easy to construct
values that are shared safely across multiple threads simultaneously.
*/
use core::fmt;
/// A lazily initialized value that implements `Deref` for `T`.
///
/// A `Lazy` takes an initialization function and permits callers from any
/// thread to access the result of that initialization function in a safe
/// manner. In effect, this permits one-time initialization of global resources
/// in a (possibly) multi-threaded program.
///
/// This type and its functionality are available even when neither the `alloc`
/// nor the `std` features are enabled. In exchange, a `Lazy` does **not**
/// guarantee that the given `create` function is called at most once. It
/// might be called multiple times. Moreover, a call to `Lazy::get` (either
/// explicitly or implicitly via `Lazy`'s `Deref` impl) may block until a `T`
/// is available.
///
/// This is very similar to `lazy_static` or `once_cell`, except it doesn't
/// guarantee that the initialization function will be run once and it works
/// in no-alloc no-std environments. With that said, if you need stronger
/// guarantees or a more flexible API, then it is recommended to use either
/// `lazy_static` or `once_cell`.
///
/// # Warning: may use a spin lock
///
/// When this crate is compiled _without_ the `alloc` feature, then this type
/// may used a spin lock internally. This can have subtle effects that may
/// be undesirable. See [Spinlocks Considered Harmful][spinharm] for a more
/// thorough treatment of this topic.
///
/// [spinharm]: https://matklad.github.io/2020/01/02/spinlocks-considered-harmful.html
///
/// # Example
///
/// This type is useful for creating regexes once, and then using them from
/// multiple threads simultaneously without worrying about synchronization.
///
/// ```
/// use regex_automata::{dfa::regex::Regex, util::lazy::Lazy, Match};
///
/// static RE: Lazy<Regex> = Lazy::new(|| Regex::new("foo[0-9]+bar").unwrap());
///
/// let expected = Some(Match::must(0, 3..14));
/// assert_eq!(expected, RE.find(b"zzzfoo12345barzzz"));
/// ```
pub struct Lazy<T, F = fn() -> T>(lazy::Lazy<T, F>);
impl<T, F> Lazy<T, F> {
/// Create a new `Lazy` value that is initialized via the given function.
///
/// The `T` type is automatically inferred from the return type of the
/// `create` function given.
pub const fn new(create: F) -> Lazy<T, F> {
Lazy(lazy::Lazy::new(create))
}
}
impl<T, F: Fn() -> T> Lazy<T, F> {
/// Return a reference to the lazily initialized value.
///
/// This routine may block if another thread is initializing a `T`.
///
/// Note that given a `x` which has type `Lazy`, this must be called via
/// `Lazy::get(x)` and not `x.get()`. This routine is defined this way
/// because `Lazy` impls `Deref` with a target of `T`.
///
/// # Panics
///
/// This panics if the `create` function inside this lazy value panics.
/// If the panic occurred in another thread, then this routine _may_ also
/// panic (but is not guaranteed to do so).
pub fn get(this: &Lazy<T, F>) -> &T {
this.0.get()
}
}
impl<T, F: Fn() -> T> core::ops::Deref for Lazy<T, F> {
type Target = T;
fn deref(&self) -> &T {
Lazy::get(self)
}
}
impl<T: fmt::Debug, F: Fn() -> T> fmt::Debug for Lazy<T, F> {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
self.0.fmt(f)
}
}
#[cfg(feature = "alloc")]
mod lazy {
use core::{
fmt,
marker::PhantomData,
sync::atomic::{AtomicPtr, Ordering},
};
use alloc::boxed::Box;
/// A non-std lazy initialized value.
///
/// This might run the initialization function more than once, but will
/// never block.
///
/// I wish I could get these semantics into the non-alloc non-std Lazy
/// type below, but I'm not sure how to do it. If you can do an alloc,
/// then the implementation becomes very simple if you don't care about
/// redundant work precisely because a pointer can be atomically swapped.
///
/// Perhaps making this approach work in the non-alloc non-std case
/// requires asking the caller for a pointer? It would make the API less
/// convenient I think.
pub(super) struct Lazy<T, F> {
data: AtomicPtr<T>,
create: F,
// This indicates to the compiler that this type can drop T. It's not
// totally clear how the absence of this marker could lead to trouble,
// but putting here doesn't have any downsides so we hedge until someone
// can from the Unsafe Working Group can tell us definitively that we
// don't need it.
//
// See: https://github.com/BurntSushi/regex-automata/issues/30
owned: PhantomData<Box<T>>,
}
// SAFETY: So long as T and &T (and F and &F) can themselves be safely
// shared among threads, so to can a Lazy<T, _>. Namely, the Lazy API only
// permits accessing a &T and initialization is free of data races. So if T
// is thread safe, then so to is Lazy<T, _>.
//
// We specifically require that T: Send in order for Lazy<T> to be Sync.
// Without that requirement, it's possible to send a T from one thread to
// another via Lazy's destructor.
//
// It's not clear whether we need F: Send+Sync for Lazy to be Sync. But
// we're conservative for now and keep both.
unsafe impl<T: Send + Sync, F: Send + Sync> Sync for Lazy<T, F> {}
impl<T, F> Lazy<T, F> {
/// Create a new alloc but non-std lazy value that is racily
/// initialized. That is, the 'create' function may be called more than
/// once.
pub(super) const fn new(create: F) -> Lazy<T, F> {
Lazy {
data: AtomicPtr::new(core::ptr::null_mut()),
create,
owned: PhantomData,
}
}
}
impl<T, F: Fn() -> T> Lazy<T, F> {
/// Get the underlying lazy value. If it hasn't been initialized
/// yet, then always attempt to initialize it (even if some other
/// thread is initializing it) and atomically attach it to this lazy
/// value before returning it.
pub(super) fn get(&self) -> &T {
if let Some(data) = self.poll() {
return data;
}
let data = (self.create)();
let mut ptr = Box::into_raw(Box::new(data));
// We attempt to stuff our initialized value into our atomic
// pointer. Upon success, we don't need to do anything. But if
// someone else beat us to the punch, then we need to make sure
// our newly created value is dropped.
let result = self.data.compare_exchange(
core::ptr::null_mut(),
ptr,
Ordering::AcqRel,
Ordering::Acquire,
);
if let Err(old) = result {
// SAFETY: We created 'ptr' via Box::into_raw above, so turning
// it back into a Box via from_raw is safe.
drop(unsafe { Box::from_raw(ptr) });
ptr = old;
}
// SAFETY: We just set the pointer above to a non-null value, even
// in the error case, and set it to a fully initialized value
// returned by 'create'.
unsafe { &*ptr }
}
/// If this lazy value has been initialized successfully, then return
/// that value. Otherwise return None immediately. This never attempts
/// to run initialization itself.
fn poll(&self) -> Option<&T> {
let ptr = self.data.load(Ordering::Acquire);
if ptr.is_null() {
return None;
}
// SAFETY: We just checked that the pointer is not null. Since it's
// not null, it must have been fully initialized by 'get' at some
// point.
Some(unsafe { &*ptr })
}
}
impl<T: fmt::Debug, F: Fn() -> T> fmt::Debug for Lazy<T, F> {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
f.debug_struct("Lazy").field("data", &self.poll()).finish()
}
}
impl<T, F> Drop for Lazy<T, F> {
fn drop(&mut self) {
let ptr = *self.data.get_mut();
if !ptr.is_null() {
// SAFETY: We just checked that 'ptr' is not null. And since
// we have exclusive access, there are no races to worry about.
drop(unsafe { Box::from_raw(ptr) });
}
}
}
}
#[cfg(not(feature = "alloc"))]
mod lazy {
use core::{
cell::Cell,
fmt,
mem::MaybeUninit,
panic::{RefUnwindSafe, UnwindSafe},
sync::atomic::{AtomicU8, Ordering},
};
/// Our 'Lazy' value can be in one of three states:
///
/// * INIT is where it starts, and also ends up back here if the
/// 'create' routine panics.
/// * BUSY is where it sits while initialization is running in exactly
/// one thread.
/// * DONE is where it sits after 'create' has completed and 'data' has
/// been fully initialized.
const LAZY_STATE_INIT: u8 = 0;
const LAZY_STATE_BUSY: u8 = 1;
const LAZY_STATE_DONE: u8 = 2;
/// A non-alloc non-std lazy initialized value.
///
/// This guarantees initialization only happens once, but uses a spinlock
/// to block in the case of simultaneous access. Blocking occurs so that
/// one thread waits while another thread initializes the value.
///
/// I would much rather have the semantics of the 'alloc' Lazy type above.
/// Namely, that we might run the initialization function more than once,
/// but we never otherwise block. However, I don't know how to do that in
/// a non-alloc non-std context.
pub(super) struct Lazy<T, F> {
state: AtomicU8,
create: Cell<Option<F>>,
data: Cell<MaybeUninit<T>>,
}
// SAFETY: So long as T and &T (and F and &F) can themselves be safely
// shared among threads, so to can a Lazy<T, _>. Namely, the Lazy API only
// permits accessing a &T and initialization is free of data races. So if T
// is thread safe, then so to is Lazy<T, _>.
unsafe impl<T: Send + Sync, F: Send + Sync> Sync for Lazy<T, F> {}
// A reference to a Lazy is unwind safe because we specifically take
// precautions to poison all accesses to a Lazy if the caller-provided
// 'create' function panics.
impl<T: UnwindSafe, F: UnwindSafe + RefUnwindSafe> RefUnwindSafe
for Lazy<T, F>
{
}
impl<T, F> Lazy<T, F> {
/// Create a new non-alloc non-std lazy value that is initialized
/// exactly once on first use using the given function.
pub(super) const fn new(create: F) -> Lazy<T, F> {
Lazy {
state: AtomicU8::new(LAZY_STATE_INIT),
create: Cell::new(Some(create)),
data: Cell::new(MaybeUninit::uninit()),
}
}
}
impl<T, F: FnOnce() -> T> Lazy<T, F> {
/// Get the underlying lazy value. If it isn't been initialized
/// yet, then either initialize it or block until some other thread
/// initializes it. If the 'create' function given to Lazy::new panics
/// (even in another thread), then this panics too.
pub(super) fn get(&self) -> &T {
// This is effectively a spinlock. We loop until we enter a DONE
// state, and if possible, initialize it ourselves. The only way
// we exit the loop is if 'create' panics, we initialize 'data' or
// some other thread initializes 'data'.
//
// Yes, I have read spinlocks considered harmful[1]. And that
// article is why this spinlock is only active when 'alloc' isn't
// enabled. I did this because I don't think there is really
// another choice without 'alloc', other than not providing this at
// all. But I think that's a big bummer.
//
// [1]: https://matklad.github.io/2020/01/02/spinlocks-considered-harmful.html
while self.state.load(Ordering::Acquire) != LAZY_STATE_DONE {
// Check if we're the first ones to get here. If so, we'll be
// the ones who initialize.
let result = self.state.compare_exchange(
LAZY_STATE_INIT,
LAZY_STATE_BUSY,
Ordering::AcqRel,
Ordering::Acquire,
);
// This means we saw the INIT state and nobody else can. So we
// must take responsibility for initializing. And by virtue of
// observing INIT, we have also told anyone else trying to
// get here that we are BUSY. If someone else sees BUSY, then
// they will spin until we finish initialization.
if let Ok(_) = result {
// Since we are guaranteed to be the only ones here, we
// know that 'create' is there... Unless someone else got
// here before us and 'create' panicked. In which case,
// 'self.create' is now 'None' and we forward the panic
// to the caller. (i.e., We implement poisoning.)
//
// SAFETY: Our use of 'self.state' guarantees that we are
// the only thread executing this line, and thus there are
// no races.
let create = unsafe {
(*self.create.as_ptr()).take().expect(
"Lazy's create function panicked, \
preventing initialization,
poisoning current thread",
)
};
let guard = Guard { state: &self.state };
// SAFETY: Our use of 'self.state' guarantees that we are
// the only thread executing this line, and thus there are
// no races.
unsafe {
(*self.data.as_ptr()).as_mut_ptr().write(create());
}
// All is well. 'self.create' ran successfully, so we
// forget the guard.
core::mem::forget(guard);
// Everything is initialized, so we can declare success.
self.state.store(LAZY_STATE_DONE, Ordering::Release);
break;
}
core::hint::spin_loop();
}
// We only get here if data is fully initialized, and thus poll
// will always return something.
self.poll().unwrap()
}
/// If this lazy value has been initialized successfully, then return
/// that value. Otherwise return None immediately. This never blocks.
fn poll(&self) -> Option<&T> {
if self.state.load(Ordering::Acquire) == LAZY_STATE_DONE {
// SAFETY: The DONE state only occurs when data has been fully
// initialized.
Some(unsafe { &*(*self.data.as_ptr()).as_ptr() })
} else {
None
}
}
}
impl<T: fmt::Debug, F: FnMut() -> T> fmt::Debug for Lazy<T, F> {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
f.debug_struct("Lazy")
.field("state", &self.state.load(Ordering::Acquire))
.field("create", &"<closure>")
.field("data", &self.poll())
.finish()
}
}
impl<T, F> Drop for Lazy<T, F> {
fn drop(&mut self) {
if *self.state.get_mut() == LAZY_STATE_DONE {
// SAFETY: state is DONE if and only if data has been fully
// initialized. At which point, it is safe to drop.
unsafe {
self.data.get_mut().assume_init_drop();
}
}
}
}
/// A guard that will reset a Lazy's state back to INIT when dropped. The
/// idea here is to 'forget' this guard on success. On failure (when a
/// panic occurs), the Drop impl runs and causes all in-progress and future
/// 'get' calls to panic. Without this guard, all in-progress and future
/// 'get' calls would spin forever. Crashing is much better than getting
/// stuck in an infinite loop.
struct Guard<'a> {
state: &'a AtomicU8,
}
impl<'a> Drop for Guard<'a> {
fn drop(&mut self) {
// We force ourselves back into an INIT state. This will in turn
// cause any future 'get' calls to attempt calling 'self.create'
// again which will in turn panic because 'self.create' will now
// be 'None'.
self.state.store(LAZY_STATE_INIT, Ordering::Release);
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn assert_send<T: Send>() {}
fn assert_sync<T: Sync>() {}
fn assert_unwind<T: core::panic::UnwindSafe>() {}
fn assert_refunwind<T: core::panic::RefUnwindSafe>() {}
#[test]
fn oibits() {
assert_send::<Lazy<u64>>();
assert_sync::<Lazy<u64>>();
assert_unwind::<Lazy<u64>>();
assert_refunwind::<Lazy<u64>>();
}
// This is a regression test because we used to rely on the inferred Sync
// impl for the Lazy type defined above (for 'alloc' mode). In the
// inferred impl, it only requires that T: Sync for Lazy<T>: Sync. But
// if we have that, we can actually make use of the fact that Lazy<T> drops
// T to create a value on one thread and drop it on another. This *should*
// require T: Send, but our missing bounds before let it sneak by.
//
// Basically, this test should not compile, so we... comment it out. We
// don't have a great way of testing compile-fail tests right now.
//
// See: https://github.com/BurntSushi/regex-automata/issues/30
/*
#[test]
fn sync_not_send() {
#[allow(dead_code)]
fn inner<T: Sync + Default>() {
let lazy = Lazy::new(move || T::default());
std::thread::scope(|scope| {
scope.spawn(|| {
Lazy::get(&lazy); // We create T in this thread
});
});
// And drop in this thread.
drop(lazy);
// So we have send a !Send type over threads. (with some more
// legwork, its possible to even sneak the value out of drop
// through thread local)
}
}
*/
}
third_party/crates/regex-automata-0.4.18/src/util/look.rs
Vendored
+2546
View File
File diff suppressed because it is too large. Load diff
third_party/crates/regex-automata-0.4.18/src/util/memchr.rs
Vendored
+93
View File
@@ -0,0 +1,93 @@
/*!
This module defines simple wrapper routines for the memchr functions from the
`memchr` crate. Basically, when the `memchr` crate is available, we use it,
otherwise we use a naive implementation which is still pretty fast.
*/
pub(crate) use self::inner::*;
#[cfg(feature = "perf-literal-substring")]
pub(super) mod inner {
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn memchr(n1: u8, haystack: &[u8]) -> Option<usize> {
memchr::memchr(n1, haystack)
}
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn memchr2(n1: u8, n2: u8, haystack: &[u8]) -> Option<usize> {
memchr::memchr2(n1, n2, haystack)
}
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn memchr3(
n1: u8,
n2: u8,
n3: u8,
haystack: &[u8],
) -> Option<usize> {
memchr::memchr3(n1, n2, n3, haystack)
}
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn memrchr(n1: u8, haystack: &[u8]) -> Option<usize> {
memchr::memrchr(n1, haystack)
}
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn memrchr2(n1: u8, n2: u8, haystack: &[u8]) -> Option<usize> {
memchr::memrchr2(n1, n2, haystack)
}
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn memrchr3(
n1: u8,
n2: u8,
n3: u8,
haystack: &[u8],
) -> Option<usize> {
memchr::memrchr3(n1, n2, n3, haystack)
}
}
#[cfg(not(feature = "perf-literal-substring"))]
pub(super) mod inner {
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn memchr(n1: u8, haystack: &[u8]) -> Option<usize> {
haystack.iter().position(|&b| b == n1)
}
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn memchr2(n1: u8, n2: u8, haystack: &[u8]) -> Option<usize> {
haystack.iter().position(|&b| b == n1 || b == n2)
}
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn memchr3(
n1: u8,
n2: u8,
n3: u8,
haystack: &[u8],
) -> Option<usize> {
haystack.iter().position(|&b| b == n1 || b == n2 || b == n3)
}
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn memrchr(n1: u8, haystack: &[u8]) -> Option<usize> {
haystack.iter().rposition(|&b| b == n1)
}
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn memrchr2(n1: u8, n2: u8, haystack: &[u8]) -> Option<usize> {
haystack.iter().rposition(|&b| b == n1 || b == n2)
}
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn memrchr3(
n1: u8,
n2: u8,
n3: u8,
haystack: &[u8],
) -> Option<usize> {
haystack.iter().rposition(|&b| b == n1 || b == n2 || b == n3)
}
}
third_party/crates/regex-automata-0.4.18/src/util/mod.rs
Vendored
+57
View File
@@ -0,0 +1,57 @@
/*!
A collection of modules that provide APIs that are useful across many regex
engines.
While one should explore the sub-modules directly to get a sense of what's
there, here are some highlights that tie the sub-modules to higher level
use cases:
* `alphabet` contains APIs that are useful if you're doing low level things
with the DFAs in this crate. For example, implementing determinization or
walking its state graph directly.
* `captures` contains APIs for dealing with capture group matches and their
mapping to "slots" used inside an NFA graph. This is also where you can find
iterators over capture group names.
* `escape` contains types for pretty-printing raw byte slices as strings.
* `iter` contains API helpers for writing regex iterators.
* `lazy` contains a no-std and no-alloc variant of `lazy_static!` and
`once_cell`.
* `look` contains APIs for matching and configuring look-around assertions.
* `pool` provides a way to reuse mutable memory allocated in a thread safe
manner.
* `prefilter` provides APIs for building prefilters and using them in searches.
* `primitives` are what you might use if you're doing lower level work on
automata, such as walking an NFA state graph.
* `syntax` provides some higher level convenience functions for interacting
with the `regex-syntax` crate.
* `wire` is useful if you're working with DFA serialization.
*/
pub mod alphabet;
#[cfg(feature = "alloc")]
pub mod captures;
pub mod escape;
#[cfg(feature = "alloc")]
pub mod interpolate;
pub mod iter;
pub mod lazy;
pub mod look;
#[cfg(feature = "alloc")]
pub mod pool;
pub mod prefilter;
pub mod primitives;
pub mod start;
#[cfg(feature = "syntax")]
pub mod syntax;
pub mod wire;
#[cfg(any(feature = "dfa-build", feature = "hybrid"))]
pub(crate) mod determinize;
pub(crate) mod empty;
pub(crate) mod int;
pub(crate) mod memchr;
pub(crate) mod search;
#[cfg(feature = "alloc")]
pub(crate) mod sparse_set;
pub(crate) mod unicode_data;
pub(crate) mod utf8;
third_party/crates/regex-automata-0.4.18/src/util/pool.rs
Vendored
+1269
View File
File diff suppressed because it is too large. Load diff
third_party/crates/regex-automata-0.4.18/src/util/prefilter/aho_corasick.rs
Vendored
+153
View File
@@ -0,0 +1,153 @@
use crate::util::{
prefilter::PrefilterI,
search::{MatchKind, Span},
};
#[derive(Clone, Debug)]
pub(crate) struct AhoCorasick {
#[cfg(not(feature = "perf-literal-multisubstring"))]
_unused: (),
#[cfg(feature = "perf-literal-multisubstring")]
ac: aho_corasick::AhoCorasick,
}
impl AhoCorasick {
pub(crate) fn new<B: AsRef<[u8]>>(
kind: MatchKind,
needles: &[B],
) -> Option<AhoCorasick> {
#[cfg(not(feature = "perf-literal-multisubstring"))]
{
None
}
#[cfg(feature = "perf-literal-multisubstring")]
{
// We used to use `aho_corasick::MatchKind::Standard` here when
// `kind` was `MatchKind::All`, but this is not correct. The
// "standard" Aho-Corasick match semantics are to report a match
// immediately as soon as it is seen, but `All` isn't like that.
// In particular, with "standard" semantics, given the needles
// "abc" and "b" and the haystack "abc," it would report a match
// at offset 1 before a match at offset 0. This is never what we
// want in the context of the regex engine, regardless of whether
// we have leftmost-first or 'all' semantics. Namely, we always
// want the leftmost match.
let ac_match_kind = match kind {
MatchKind::LeftmostFirst | MatchKind::All => {
aho_corasick::MatchKind::LeftmostFirst
}
};
// This is kind of just an arbitrary number, but basically, if we
// have a small enough set of literals, then we try to use the VERY
// memory hungry DFA. Otherwise, we wimp out and use an NFA. The
// upshot is that the NFA is quite lean and decently fast. Faster
// than a naive Aho-Corasick NFA anyway.
let ac_kind = if needles.len() <= 500 {
aho_corasick::AhoCorasickKind::DFA
} else {
aho_corasick::AhoCorasickKind::ContiguousNFA
};
let result = aho_corasick::AhoCorasick::builder()
.kind(Some(ac_kind))
.match_kind(ac_match_kind)
.start_kind(aho_corasick::StartKind::Both)
// We try to handle all of the prefilter cases in the super
// module, and only use Aho-Corasick for the actual automaton.
// The aho-corasick crate does have some extra prefilters,
// namely, looking for rare bytes to feed to memchr{,2,3}
// instead of just the first byte. If we end up wanting
// those---and they are somewhat tricky to implement---then
// we could port them to this crate.
//
// The main reason for doing things this way is so we have a
// complete and easy to understand picture of which prefilters
// are available and how they work. Otherwise it seems too
// easy to get into a situation where we have a prefilter
// layered on top of prefilter, and that might have unintended
// consequences.
.prefilter(false)
.build(needles);
let ac = match result {
Ok(ac) => ac,
Err(_err) => {
debug!("aho-corasick prefilter failed to build: {_err}");
return None;
}
};
Some(AhoCorasick { ac })
}
}
}
impl PrefilterI for AhoCorasick {
fn name(&self) -> &'static str {
"aho-corasick"
}
fn find(&self, haystack: &[u8], span: Span) -> Option<Span> {
#[cfg(not(feature = "perf-literal-multisubstring"))]
{
unreachable!()
}
#[cfg(feature = "perf-literal-multisubstring")]
{
let input =
aho_corasick::Input::new(haystack).span(span.start..span.end);
self.ac
.find(input)
.map(|m| Span { start: m.start(), end: m.end() })
}
}
fn prefix(&self, haystack: &[u8], span: Span) -> Option<Span> {
#[cfg(not(feature = "perf-literal-multisubstring"))]
{
unreachable!()
}
#[cfg(feature = "perf-literal-multisubstring")]
{
let input = aho_corasick::Input::new(haystack)
.anchored(aho_corasick::Anchored::Yes)
.span(span.start..span.end);
self.ac
.find(input)
.map(|m| Span { start: m.start(), end: m.end() })
}
}
fn memory_usage(&self) -> usize {
#[cfg(not(feature = "perf-literal-multisubstring"))]
{
unreachable!()
}
#[cfg(feature = "perf-literal-multisubstring")]
{
self.ac.memory_usage()
}
}
fn is_fast(&self) -> bool {
#[cfg(not(feature = "perf-literal-multisubstring"))]
{
unreachable!()
}
#[cfg(feature = "perf-literal-multisubstring")]
{
// Aho-Corasick is never considered "fast" because it's never
// going to be even close to an order of magnitude faster than the
// regex engine itself (assuming a DFA is used). In fact, it is
// usually slower. The magic of Aho-Corasick is that it can search
// a *large* number of literals with a relatively small amount of
// memory. The regex engines are far more wasteful.
//
// Aho-Corasick may be "fast" when the regex engine corresponds
// to, say, the PikeVM. That happens when the lazy DFA couldn't be
// built or used for some reason. But in these cases, the regex
// itself is likely quite big and we're probably hosed no matter
// what we do. (In this case, the best bet is for the caller to
// increase some of the memory limits on the hybrid cache capacity
// and hope that's enough.)
false
}
}
}
third_party/crates/regex-automata-0.4.18/src/util/prefilter/byteset.rs
Vendored
+62
View File
@@ -0,0 +1,62 @@
use crate::util::{
prefilter::PrefilterI,
search::{MatchKind, Span},
};
#[derive(Clone, Debug)]
pub(crate) struct ByteSet([bool; 256]);
impl ByteSet {
pub(crate) fn new<B: AsRef<[u8]>>(
_kind: MatchKind,
needles: &[B],
) -> Option<ByteSet> {
#[cfg(not(feature = "perf-literal-multisubstring"))]
{
None
}
#[cfg(feature = "perf-literal-multisubstring")]
{
let mut set = [false; 256];
for needle in needles.iter() {
let needle = needle.as_ref();
if needle.len() != 1 {
return None;
}
set[usize::from(needle[0])] = true;
}
Some(ByteSet(set))
}
}
}
impl PrefilterI for ByteSet {
fn name(&self) -> &'static str {
"byteset"
}
fn find(&self, haystack: &[u8], span: Span) -> Option<Span> {
haystack[span].iter().position(|&b| self.0[usize::from(b)]).map(|i| {
let start = span.start + i;
let end = start + 1;
Span { start, end }
})
}
fn prefix(&self, haystack: &[u8], span: Span) -> Option<Span> {
let b = *haystack.get(span.start)?;
if self.0[usize::from(b)] {
Some(Span { start: span.start, end: span.start + 1 })
} else {
None
}
}
fn memory_usage(&self) -> usize {
0
}
fn is_fast(&self) -> bool {
false
}
}
third_party/crates/regex-automata-0.4.18/src/util/prefilter/memchr.rs
Vendored
+198
View File
@@ -0,0 +1,198 @@
use crate::util::{
prefilter::PrefilterI,
search::{MatchKind, Span},
};
#[derive(Clone, Debug)]
pub(crate) struct Memchr(u8);
impl Memchr {
pub(crate) fn new<B: AsRef<[u8]>>(
_kind: MatchKind,
needles: &[B],
) -> Option<Memchr> {
#[cfg(not(feature = "perf-literal-substring"))]
{
None
}
#[cfg(feature = "perf-literal-substring")]
{
if needles.len() != 1 {
return None;
}
if needles[0].as_ref().len() != 1 {
return None;
}
Some(Memchr(needles[0].as_ref()[0]))
}
}
}
impl PrefilterI for Memchr {
fn name(&self) -> &'static str {
"memchr"
}
fn find(&self, haystack: &[u8], span: Span) -> Option<Span> {
#[cfg(not(feature = "perf-literal-substring"))]
{
unreachable!()
}
#[cfg(feature = "perf-literal-substring")]
{
memchr::memchr(self.0, &haystack[span]).map(|i| {
let start = span.start + i;
let end = start + 1;
Span { start, end }
})
}
}
fn prefix(&self, haystack: &[u8], span: Span) -> Option<Span> {
let b = *haystack.get(span.start)?;
if self.0 == b {
Some(Span { start: span.start, end: span.start + 1 })
} else {
None
}
}
fn memory_usage(&self) -> usize {
0
}
fn is_fast(&self) -> bool {
true
}
}
#[derive(Clone, Debug)]
pub(crate) struct Memchr2(u8, u8);
impl Memchr2 {
pub(crate) fn new<B: AsRef<[u8]>>(
_kind: MatchKind,
needles: &[B],
) -> Option<Memchr2> {
#[cfg(not(feature = "perf-literal-substring"))]
{
None
}
#[cfg(feature = "perf-literal-substring")]
{
if needles.len() != 2 {
return None;
}
if !needles.iter().all(|n| n.as_ref().len() == 1) {
return None;
}
let b1 = needles[0].as_ref()[0];
let b2 = needles[1].as_ref()[0];
Some(Memchr2(b1, b2))
}
}
}
impl PrefilterI for Memchr2 {
fn name(&self) -> &'static str {
"memchr2"
}
fn find(&self, haystack: &[u8], span: Span) -> Option<Span> {
#[cfg(not(feature = "perf-literal-substring"))]
{
unreachable!()
}
#[cfg(feature = "perf-literal-substring")]
{
memchr::memchr2(self.0, self.1, &haystack[span]).map(|i| {
let start = span.start + i;
let end = start + 1;
Span { start, end }
})
}
}
fn prefix(&self, haystack: &[u8], span: Span) -> Option<Span> {
let b = *haystack.get(span.start)?;
if self.0 == b || self.1 == b {
Some(Span { start: span.start, end: span.start + 1 })
} else {
None
}
}
fn memory_usage(&self) -> usize {
0
}
fn is_fast(&self) -> bool {
true
}
}
#[derive(Clone, Debug)]
pub(crate) struct Memchr3(u8, u8, u8);
impl Memchr3 {
pub(crate) fn new<B: AsRef<[u8]>>(
_kind: MatchKind,
needles: &[B],
) -> Option<Memchr3> {
#[cfg(not(feature = "perf-literal-substring"))]
{
None
}
#[cfg(feature = "perf-literal-substring")]
{
if needles.len() != 3 {
return None;
}
if !needles.iter().all(|n| n.as_ref().len() == 1) {
return None;
}
let b1 = needles[0].as_ref()[0];
let b2 = needles[1].as_ref()[0];
let b3 = needles[2].as_ref()[0];
Some(Memchr3(b1, b2, b3))
}
}
}
impl PrefilterI for Memchr3 {
fn name(&self) -> &'static str {
"memchr3"
}
fn find(&self, haystack: &[u8], span: Span) -> Option<Span> {
#[cfg(not(feature = "perf-literal-substring"))]
{
unreachable!()
}
#[cfg(feature = "perf-literal-substring")]
{
memchr::memchr3(self.0, self.1, self.2, &haystack[span]).map(|i| {
let start = span.start + i;
let end = start + 1;
Span { start, end }
})
}
}
fn prefix(&self, haystack: &[u8], span: Span) -> Option<Span> {
let b = *haystack.get(span.start)?;
if self.0 == b || self.1 == b || self.2 == b {
Some(Span { start: span.start, end: span.start + 1 })
} else {
None
}
}
fn memory_usage(&self) -> usize {
0
}
fn is_fast(&self) -> bool {
true
}
}
third_party/crates/regex-automata-0.4.18/src/util/prefilter/memmem.rs
Vendored
+92
View File
@@ -0,0 +1,92 @@
use crate::util::{
prefilter::PrefilterI,
search::{MatchKind, Span},
};
#[derive(Clone, Debug)]
pub(crate) struct Memmem {
#[cfg(not(all(feature = "std", feature = "perf-literal-substring")))]
_unused: (),
#[cfg(all(feature = "std", feature = "perf-literal-substring"))]
finder: memchr::memmem::Finder<'static>,
}
impl Memmem {
pub(crate) fn new<B: AsRef<[u8]>>(
_kind: MatchKind,
needles: &[B],
) -> Option<Memmem> {
#[cfg(not(all(feature = "std", feature = "perf-literal-substring")))]
{
None
}
#[cfg(all(feature = "std", feature = "perf-literal-substring"))]
{
if needles.len() != 1 {
return None;
}
let needle = needles[0].as_ref();
let finder = memchr::memmem::Finder::new(needle).into_owned();
Some(Memmem { finder })
}
}
}
impl PrefilterI for Memmem {
fn name(&self) -> &'static str {
"memmem"
}
fn find(&self, haystack: &[u8], span: Span) -> Option<Span> {
#[cfg(not(all(feature = "std", feature = "perf-literal-substring")))]
{
unreachable!()
}
#[cfg(all(feature = "std", feature = "perf-literal-substring"))]
{
self.finder.find(&haystack[span]).map(|i| {
let start = span.start + i;
let end = start + self.finder.needle().len();
Span { start, end }
})
}
}
fn prefix(&self, haystack: &[u8], span: Span) -> Option<Span> {
#[cfg(not(all(feature = "std", feature = "perf-literal-substring")))]
{
unreachable!()
}
#[cfg(all(feature = "std", feature = "perf-literal-substring"))]
{
let needle = self.finder.needle();
if haystack[span].starts_with(needle) {
Some(Span { end: span.start + needle.len(), ..span })
} else {
None
}
}
}
fn memory_usage(&self) -> usize {
#[cfg(not(all(feature = "std", feature = "perf-literal-substring")))]
{
unreachable!()
}
#[cfg(all(feature = "std", feature = "perf-literal-substring"))]
{
self.finder.needle().len()
}
}
fn is_fast(&self) -> bool {
#[cfg(not(all(feature = "std", feature = "perf-literal-substring")))]
{
unreachable!()
}
#[cfg(all(feature = "std", feature = "perf-literal-substring"))]
{
true
}
}
}
third_party/crates/regex-automata-0.4.18/src/util/prefilter/mod.rs
Vendored
+726
View File
@@ -0,0 +1,726 @@
/*!
Defines a prefilter for accelerating regex searches.
A prefilter can be created by building a [`Prefilter`] value.
A prefilter represents one of the most important optimizations available for
accelerating regex searches. The idea of a prefilter is to very quickly find
candidate locations in a haystack where a regex _could_ match. Once a candidate
is found, it is then intended for the regex engine to run at that position to
determine whether the candidate is a match or a false positive.
In the aforementioned description of the prefilter optimization also lay its
demise. Namely, if a prefilter has a high false positive rate and it produces
lots of candidates, then a prefilter can overall make a regex search slower.
It can run more slowly because more time is spent ping-ponging between the
prefilter search and the regex engine attempting to confirm each candidate as
a match. This ping-ponging has overhead that adds up, and is exacerbated by
a high false positive rate.
Nevertheless, the optimization is still generally worth performing in most
cases. Particularly given just how much throughput can be improved. (It is not
uncommon for prefilter optimizations to improve throughput by one or two orders
of magnitude.)
Typically a prefilter is used to find occurrences of literal prefixes from a
regex pattern, but this isn't required. A prefilter can be used to look for
suffixes or even inner literals.
Note that as of now, prefilters throw away information about which pattern
each literal comes from. In other words, when a prefilter finds a match,
there's no way to know which pattern (or patterns) it came from. Therefore,
in order to confirm a match, you'll have to check all of the patterns by
running the full regex engine.
*/
mod aho_corasick;
mod byteset;
mod memchr;
mod memmem;
mod teddy;
use core::{
borrow::Borrow,
fmt::Debug,
panic::{RefUnwindSafe, UnwindSafe},
};
#[cfg(feature = "alloc")]
use alloc::sync::Arc;
#[cfg(feature = "syntax")]
use regex_syntax::hir::{literal, Hir};
use crate::util::search::{MatchKind, Span};
pub(crate) use crate::util::prefilter::{
aho_corasick::AhoCorasick,
byteset::ByteSet,
memchr::{Memchr, Memchr2, Memchr3},
memmem::Memmem,
teddy::Teddy,
};
/// A prefilter for accelerating regex searches.
///
/// If you already have your literals that you want to search with,
/// then the vanilla [`Prefilter::new`] constructor is for you. But
/// if you have an [`Hir`] value from the `regex-syntax` crate, then
/// [`Prefilter::from_hir_prefix`] might be more convenient. Namely, it uses
/// the [`regex-syntax::hir::literal`](regex_syntax::hir::literal) module to
/// extract literal prefixes for you, optimize them and then select and build a
/// prefilter matcher.
///
/// A prefilter must have **zero false negatives**. However, by its very
/// nature, it may produce false positives. That is, a prefilter will never
/// skip over a position in the haystack that corresponds to a match of the
/// original regex pattern, but it *may* produce a match for a position
/// in the haystack that does *not* correspond to a match of the original
/// regex pattern. If you use either the [`Prefilter::from_hir_prefix`] or
/// [`Prefilter::from_hirs_prefix`] constructors, then this guarantee is
/// upheld for you automatically. This guarantee is not preserved if you use
/// [`Prefilter::new`] though, since it is up to the caller to provide correct
/// literal strings with respect to the original regex pattern.
///
/// # Cloning
///
/// It is an API guarantee that cloning a prefilter is cheap. That is, cloning
/// it will not duplicate whatever heap memory is used to represent the
/// underlying matcher.
///
/// # Example
///
/// This example shows how to attach a `Prefilter` to the
/// [`PikeVM`](crate::nfa::thompson::pikevm::PikeVM) in order to accelerate
/// searches.
///
/// ```
/// use regex_automata::{
/// nfa::thompson::pikevm::PikeVM,
/// util::prefilter::Prefilter,
/// Match, MatchKind,
/// };
///
/// let pre = Prefilter::new(MatchKind::LeftmostFirst, &["Bruce "])
/// .expect("a prefilter");
/// let re = PikeVM::builder()
/// .configure(PikeVM::config().prefilter(Some(pre)))
/// .build(r"Bruce \w+")?;
/// let mut cache = re.create_cache();
/// assert_eq!(
/// Some(Match::must(0, 6..23)),
/// re.find(&mut cache, "Hello Bruce Springsteen!"),
/// );
/// # Ok::<(), Box<dyn std::error::Error>>(())
/// ```
///
/// But note that if you get your prefilter incorrect, it could lead to an
/// incorrect result!
///
/// ```
/// use regex_automata::{
/// nfa::thompson::pikevm::PikeVM,
/// util::prefilter::Prefilter,
/// Match, MatchKind,
/// };
///
/// // This prefilter is wrong!
/// let pre = Prefilter::new(MatchKind::LeftmostFirst, &["Patti "])
/// .expect("a prefilter");
/// let re = PikeVM::builder()
/// .configure(PikeVM::config().prefilter(Some(pre)))
/// .build(r"Bruce \w+")?;
/// let mut cache = re.create_cache();
/// // We find no match even though the regex does match.
/// assert_eq!(
/// None,
/// re.find(&mut cache, "Hello Bruce Springsteen!"),
/// );
/// # Ok::<(), Box<dyn std::error::Error>>(())
/// ```
#[derive(Clone, Debug)]
pub struct Prefilter {
#[cfg(not(feature = "alloc"))]
_unused: (),
#[cfg(feature = "alloc")]
pre: Arc<dyn PrefilterI>,
#[cfg(feature = "alloc")]
is_fast: bool,
#[cfg(feature = "alloc")]
max_needle_len: usize,
}
impl Prefilter {
/// Create a new prefilter from a sequence of needles and a corresponding
/// match semantics.
///
/// This may return `None` for a variety of reasons, for example, if
/// a suitable prefilter could not be constructed. That might occur
/// if they are unavailable (e.g., the `perf-literal-substring` and
/// `perf-literal-multisubstring` features aren't enabled), or it might
/// occur because of heuristics or other artifacts of how the prefilter
/// works.
///
/// Note that if you have an [`Hir`] expression, it may be more convenient
/// to use [`Prefilter::from_hir_prefix`]. It will automatically handle the
/// task of extracting prefix literals for you.
///
/// # Example
///
/// This example shows how match semantics can impact the matching
/// algorithm used by the prefilter. For this reason, it is important to
/// ensure that the match semantics given here are consistent with the
/// match semantics intended for the regular expression that the literals
/// were extracted from.
///
/// ```
/// use regex_automata::{
/// util::{prefilter::Prefilter, syntax},
/// MatchKind, Span,
/// };
///
/// let hay = "Hello samwise";
///
/// // With leftmost-first, we find 'samwise' here because it comes
/// // before 'sam' in the sequence we give it..
/// let pre = Prefilter::new(MatchKind::LeftmostFirst, &["samwise", "sam"])
/// .expect("a prefilter");
/// assert_eq!(
/// Some(Span::from(6..13)),
/// pre.find(hay.as_bytes(), Span::from(0..hay.len())),
/// );
/// // Still with leftmost-first but with the literals reverse, now 'sam'
/// // will match instead!
/// let pre = Prefilter::new(MatchKind::LeftmostFirst, &["sam", "samwise"])
/// .expect("a prefilter");
/// assert_eq!(
/// Some(Span::from(6..9)),
/// pre.find(hay.as_bytes(), Span::from(0..hay.len())),
/// );
///
/// # Ok::<(), Box<dyn std::error::Error>>(())
/// ```
pub fn new<B: AsRef<[u8]>>(
kind: MatchKind,
needles: &[B],
) -> Option<Prefilter> {
Choice::new(kind, needles).and_then(|choice| {
let max_needle_len =
needles.iter().map(|b| b.as_ref().len()).max().unwrap_or(0);
Prefilter::from_choice(choice, max_needle_len)
})
}
/// This turns a prefilter selection into a `Prefilter`. That is, in turns
/// the enum given into a trait object.
fn from_choice(
choice: Choice,
max_needle_len: usize,
) -> Option<Prefilter> {
#[cfg(not(feature = "alloc"))]
{
None
}
#[cfg(feature = "alloc")]
{
let pre: Arc<dyn PrefilterI> = match choice {
Choice::Memchr(p) => Arc::new(p),
Choice::Memchr2(p) => Arc::new(p),
Choice::Memchr3(p) => Arc::new(p),
Choice::Memmem(p) => Arc::new(p),
Choice::Teddy(p) => Arc::new(p),
Choice::ByteSet(p) => Arc::new(p),
Choice::AhoCorasick(p) => Arc::new(p),
};
let is_fast = pre.is_fast();
Some(Prefilter { pre, is_fast, max_needle_len })
}
}
/// This attempts to extract prefixes from the given `Hir` expression for
/// the given match semantics, and if possible, builds a prefilter for
/// them.
///
/// # Example
///
/// This example shows how to build a prefilter directly from an [`Hir`]
/// expression, and use to find an occurrence of a prefix from the regex
/// pattern.
///
/// ```
/// use regex_automata::{
/// util::{prefilter::Prefilter, syntax},
/// MatchKind, Span,
/// };
///
/// let hir = syntax::parse(r"(Bruce|Patti) \w+")?;
/// let pre = Prefilter::from_hir_prefix(MatchKind::LeftmostFirst, &hir)
/// .expect("a prefilter");
/// let hay = "Hello Patti Scialfa!";
/// assert_eq!(
/// Some(Span::from(6..12)),
/// pre.find(hay.as_bytes(), Span::from(0..hay.len())),
/// );
///
/// # Ok::<(), Box<dyn std::error::Error>>(())
/// ```
#[cfg(feature = "syntax")]
pub fn from_hir_prefix(kind: MatchKind, hir: &Hir) -> Option<Prefilter> {
Prefilter::from_hirs_prefix(kind, &[hir])
}
/// This attempts to extract prefixes from the given `Hir` expressions for
/// the given match semantics, and if possible, builds a prefilter for
/// them.
///
/// Note that as of now, prefilters throw away information about which
/// pattern each literal comes from. In other words, when a prefilter finds
/// a match, there's no way to know which pattern (or patterns) it came
/// from. Therefore, in order to confirm a match, you'll have to check all
/// of the patterns by running the full regex engine.
///
/// # Example
///
/// This example shows how to build a prefilter directly from multiple
/// `Hir` expressions expression, and use it to find an occurrence of a
/// prefix from the regex patterns.
///
/// ```
/// use regex_automata::{
/// util::{prefilter::Prefilter, syntax},
/// MatchKind, Span,
/// };
///
/// let hirs = syntax::parse_many(&[
/// r"(Bruce|Patti) \w+",
/// r"Mrs?\. Doubtfire",
/// ])?;
/// let pre = Prefilter::from_hirs_prefix(MatchKind::LeftmostFirst, &hirs)
/// .expect("a prefilter");
/// let hay = "Hello Mrs. Doubtfire";
/// assert_eq!(
/// Some(Span::from(6..20)),
/// pre.find(hay.as_bytes(), Span::from(0..hay.len())),
/// );
///
/// # Ok::<(), Box<dyn std::error::Error>>(())
/// ```
#[cfg(feature = "syntax")]
pub fn from_hirs_prefix<H: Borrow<Hir>>(
kind: MatchKind,
hirs: &[H],
) -> Option<Prefilter> {
prefixes(kind, hirs)
.literals()
.and_then(|lits| Prefilter::new(kind, lits))
}
/// Run this prefilter on `haystack[span.start..end]` and return a matching
/// span if one exists.
///
/// The span returned is guaranteed to have a start position greater than
/// or equal to the one given, and an end position less than or equal to
/// the one given.
///
/// # Example
///
/// This example shows how to build a prefilter directly from an [`Hir`]
/// expression, and use it to find an occurrence of a prefix from the regex
/// pattern.
///
/// ```
/// use regex_automata::{
/// util::{prefilter::Prefilter, syntax},
/// MatchKind, Span,
/// };
///
/// let hir = syntax::parse(r"Bruce \w+")?;
/// let pre = Prefilter::from_hir_prefix(MatchKind::LeftmostFirst, &hir)
/// .expect("a prefilter");
/// let hay = "Hello Bruce Springsteen!";
/// assert_eq!(
/// Some(Span::from(6..12)),
/// pre.find(hay.as_bytes(), Span::from(0..hay.len())),
/// );
///
/// # Ok::<(), Box<dyn std::error::Error>>(())
/// ```
#[inline]
pub fn find(&self, haystack: &[u8], span: Span) -> Option<Span> {
#[cfg(not(feature = "alloc"))]
{
unreachable!()
}
#[cfg(feature = "alloc")]
{
self.pre.find(haystack, span)
}
}
/// Returns the span of a prefix of `haystack[span.start..span.end]` if
/// the prefilter matches.
///
/// The span returned is guaranteed to have a start position equivalent to
/// the one given, and an end position less than or equal to the one given.
///
/// # Example
///
/// This example shows how to build a prefilter directly from an [`Hir`]
/// expression, and use it to find an occurrence of a prefix from the regex
/// pattern that begins at the start of a haystack only.
///
/// ```
/// use regex_automata::{
/// util::{prefilter::Prefilter, syntax},
/// MatchKind, Span,
/// };
///
/// let hir = syntax::parse(r"Bruce \w+")?;
/// let pre = Prefilter::from_hir_prefix(MatchKind::LeftmostFirst, &hir)
/// .expect("a prefilter");
/// let hay = "Hello Bruce Springsteen!";
/// // Nothing is found here because 'Bruce' does
/// // not occur at the beginning of our search.
/// assert_eq!(
/// None,
/// pre.prefix(hay.as_bytes(), Span::from(0..hay.len())),
/// );
/// // But if we change where we start the search
/// // to begin where 'Bruce ' begins, then a
/// // match will be found.
/// assert_eq!(
/// Some(Span::from(6..12)),
/// pre.prefix(hay.as_bytes(), Span::from(6..hay.len())),
/// );
///
/// # Ok::<(), Box<dyn std::error::Error>>(())
/// ```
#[inline]
pub fn prefix(&self, haystack: &[u8], span: Span) -> Option<Span> {
#[cfg(not(feature = "alloc"))]
{
unreachable!()
}
#[cfg(feature = "alloc")]
{
self.pre.prefix(haystack, span)
}
}
/// Returns the heap memory, in bytes, used by the underlying prefilter.
#[inline]
pub fn memory_usage(&self) -> usize {
#[cfg(not(feature = "alloc"))]
{
unreachable!()
}
#[cfg(feature = "alloc")]
{
self.pre.memory_usage()
}
}
/// Return the length of the longest needle
/// in this Prefilter
#[inline]
pub fn max_needle_len(&self) -> usize {
#[cfg(not(feature = "alloc"))]
{
unreachable!()
}
#[cfg(feature = "alloc")]
{
self.max_needle_len
}
}
/// Implementations might return true here if they believe themselves to
/// be "fast." The concept of "fast" is deliberately left vague, but in
/// practice this usually corresponds to whether it's believed that SIMD
/// will be used.
///
/// Why do we care about this? Well, some prefilter tricks tend to come
/// with their own bits of overhead, and so might only make sense if we
/// know that a scan will be *much* faster than the regex engine itself.
/// Otherwise, the trick may not be worth doing. Whether something is
/// "much" faster than the regex engine generally boils down to whether
/// SIMD is used. (But not always. Even a SIMD matcher with a high false
/// positive rate can become quite slow.)
///
/// Even if this returns true, it is still possible for the prefilter to
/// be "slow." Remember, prefilters are just heuristics. We can't really
/// *know* a prefilter will be fast without actually trying the prefilter.
/// (Which of course we cannot afford to do.)
#[inline]
pub fn is_fast(&self) -> bool {
#[cfg(not(feature = "alloc"))]
{
unreachable!()
}
#[cfg(feature = "alloc")]
{
self.is_fast
}
}
}
/// A trait for abstracting over prefilters. Basically, a prefilter is
/// something that do an unanchored *and* an anchored search in a haystack
/// within a given span.
///
/// This exists pretty much only so that we can use prefilters as a trait
/// object (which is what `Prefilter` is). If we ever move off of trait objects
/// and to an enum, then it's likely this trait could be removed.
pub(crate) trait PrefilterI:
Debug + Send + Sync + RefUnwindSafe + UnwindSafe + 'static
{
/// Return the name of this prefilter.
fn name(&self) -> &'static str;
/// Run this prefilter on `haystack[span.start..end]` and return a matching
/// span if one exists.
///
/// The span returned is guaranteed to have a start position greater than
/// or equal to the one given, and an end position less than or equal to
/// the one given.
fn find(&self, haystack: &[u8], span: Span) -> Option<Span>;
/// Returns the span of a prefix of `haystack[span.start..span.end]` if
/// the prefilter matches.
///
/// The span returned is guaranteed to have a start position equivalent to
/// the one given, and an end position less than or equal to the one given.
fn prefix(&self, haystack: &[u8], span: Span) -> Option<Span>;
/// Returns the heap memory, in bytes, used by the underlying prefilter.
fn memory_usage(&self) -> usize;
/// Implementations might return true here if they believe themselves to
/// be "fast." See [`Prefilter::is_fast`] for more details.
fn is_fast(&self) -> bool;
}
#[cfg(feature = "alloc")]
impl<P: PrefilterI + ?Sized> PrefilterI for Arc<P> {
fn name(&self) -> &'static str {
(**self).name()
}
#[cfg_attr(feature = "perf-inline", inline(always))]
fn find(&self, haystack: &[u8], span: Span) -> Option<Span> {
(**self).find(haystack, span)
}
#[cfg_attr(feature = "perf-inline", inline(always))]
fn prefix(&self, haystack: &[u8], span: Span) -> Option<Span> {
(**self).prefix(haystack, span)
}
#[cfg_attr(feature = "perf-inline", inline(always))]
fn memory_usage(&self) -> usize {
(**self).memory_usage()
}
#[cfg_attr(feature = "perf-inline", inline(always))]
fn is_fast(&self) -> bool {
(&**self).is_fast()
}
}
/// A type that encapsulates the selection of a prefilter algorithm from a
/// sequence of needles.
///
/// The existence of this type is a little tricky, because we don't (currently)
/// use it for performing a search. Instead, we really only consume it by
/// converting the underlying prefilter into a trait object, whether that be
/// `dyn PrefilterI` or `dyn Strategy` (for the meta regex engine). In order
/// to avoid re-copying the prefilter selection logic, we isolate it here, and
/// then force anything downstream that wants to convert it to a trait object
/// to do trivial case analysis on it.
///
/// One wonders whether we *should* use an enum instead of a trait object.
/// At time of writing, I chose trait objects based on instinct because 1) I
/// knew I wasn't going to inline anything and 2) there would potentially be
/// many different choices. However, as of time of writing, I haven't actually
/// compared the trait object approach to the enum approach. That probably
/// should be litigated, but I ran out of steam.
///
/// Note that if the `alloc` feature is disabled, then values of this type
/// are (and should) never be constructed. Also, in practice, for any of the
/// prefilters to be selected, you'll need at least one of the `perf-literal-*`
/// features enabled.
#[derive(Clone, Debug)]
pub(crate) enum Choice {
Memchr(Memchr),
Memchr2(Memchr2),
Memchr3(Memchr3),
Memmem(Memmem),
Teddy(Teddy),
ByteSet(ByteSet),
AhoCorasick(AhoCorasick),
}
impl Choice {
/// Select what is believed to be the best prefilter algorithm for the
/// match semantics and sequence of needles given.
///
/// This selection algorithm uses the needles as given without any
/// modification. For example, if `[bar]` is given, then this doesn't
/// try to select `memchr` for `b`. Instead, it would select `memmem`
/// for `bar`. If callers would want `memchr` selected for `[bar]`, then
/// callers should massages the literals themselves. That is, callers are
/// responsible for heuristics surrounding which sequence of literals is
/// best.
///
/// What this selection algorithm does is attempt to use the fastest
/// prefilter that works for the literals given. So if `[a, b]`, is given,
/// then `memchr2` is selected.
///
/// Of course, which prefilter is selected is also subject to what
/// is available. For example, if `alloc` isn't enabled, then
/// that limits which prefilters can be selected. Similarly, if
/// `perf-literal-substring` isn't enabled, then nothing from the `memchr`
/// crate can be returned.
pub(crate) fn new<B: AsRef<[u8]>>(
kind: MatchKind,
needles: &[B],
) -> Option<Choice> {
// An empty set means the regex matches nothing, so no sense in
// building a prefilter.
if needles.len() == 0 {
debug!("prefilter building failed: found empty set of literals");
return None;
}
// If the regex can match the empty string, then the prefilter
// will by definition match at every position. This is obviously
// completely ineffective.
if needles.iter().any(|n| n.as_ref().is_empty()) {
debug!("prefilter building failed: literals match empty string");
return None;
}
// BREADCRUMBS: Perhaps the literal optimizer should special case
// sequences of length two or three if the leading bytes of each are
// "rare"? Or perhaps, if there are two or three total possible leading
// bytes, regardless of the number of literals, and all are rare...
// Then well, perhaps we should use memchr2 or memchr3 in those cases?
if let Some(pre) = Memchr::new(kind, needles) {
debug!("prefilter built: memchr");
return Some(Choice::Memchr(pre));
}
if let Some(pre) = Memchr2::new(kind, needles) {
debug!("prefilter built: memchr2");
return Some(Choice::Memchr2(pre));
}
if let Some(pre) = Memchr3::new(kind, needles) {
debug!("prefilter built: memchr3");
return Some(Choice::Memchr3(pre));
}
if let Some(pre) = Memmem::new(kind, needles) {
debug!("prefilter built: memmem");
return Some(Choice::Memmem(pre));
}
if let Some(pre) = Teddy::new(kind, needles) {
debug!("prefilter built: teddy");
return Some(Choice::Teddy(pre));
}
if let Some(pre) = ByteSet::new(kind, needles) {
debug!("prefilter built: byteset");
return Some(Choice::ByteSet(pre));
}
if let Some(pre) = AhoCorasick::new(kind, needles) {
debug!("prefilter built: aho-corasick");
return Some(Choice::AhoCorasick(pre));
}
debug!("prefilter building failed: no strategy could be found");
None
}
}
/// Extracts all of the prefix literals from the given HIR expressions into a
/// single `Seq`. The literals in the sequence are ordered with respect to the
/// order of the given HIR expressions and consistent with the match semantics
/// given.
///
/// The sequence returned is "optimized." That is, they may be shrunk or even
/// truncated according to heuristics with the intent of making them more
/// useful as a prefilter. (Which translates to both using faster algorithms
/// and minimizing the false positive rate.)
///
/// Note that this erases any connection between the literals and which pattern
/// (or patterns) they came from.
///
/// The match kind given must correspond to the match semantics of the regex
/// that is represented by the HIRs given. The match semantics may change the
/// literal sequence returned.
#[cfg(feature = "syntax")]
pub(crate) fn prefixes<H>(kind: MatchKind, hirs: &[H]) -> literal::Seq
where
H: core::borrow::Borrow<Hir>,
{
let mut extractor = literal::Extractor::new();
extractor.kind(literal::ExtractKind::Prefix);
let mut prefixes = literal::Seq::empty();
for hir in hirs {
prefixes.union(&mut extractor.extract(hir.borrow()));
}
debug!(
"prefixes (len={:?}, exact={:?}) extracted before optimization: {:?}",
prefixes.len(),
prefixes.is_exact(),
prefixes
);
match kind {
MatchKind::All => {
prefixes.sort();
prefixes.dedup();
}
MatchKind::LeftmostFirst => {
prefixes.optimize_for_prefix_by_preference();
}
}
debug!(
"prefixes (len={:?}, exact={:?}) extracted after optimization: {:?}",
prefixes.len(),
prefixes.is_exact(),
prefixes
);
prefixes
}
/// Like `prefixes`, but for all suffixes of all matches for the given HIRs.
#[cfg(feature = "syntax")]
pub(crate) fn suffixes<H>(kind: MatchKind, hirs: &[H]) -> literal::Seq
where
H: core::borrow::Borrow<Hir>,
{
let mut extractor = literal::Extractor::new();
extractor.kind(literal::ExtractKind::Suffix);
let mut suffixes = literal::Seq::empty();
for hir in hirs {
suffixes.union(&mut extractor.extract(hir.borrow()));
}
debug!(
"suffixes (len={:?}, exact={:?}) extracted before optimization: {:?}",
suffixes.len(),
suffixes.is_exact(),
suffixes
);
match kind {
MatchKind::All => {
suffixes.sort();
suffixes.dedup();
}
MatchKind::LeftmostFirst => {
suffixes.optimize_for_suffix_by_preference();
}
}
debug!(
"suffixes (len={:?}, exact={:?}) extracted after optimization: {:?}",
suffixes.len(),
suffixes.is_exact(),
suffixes
);
suffixes
}
third_party/crates/regex-automata-0.4.18/src/util/prefilter/teddy.rs
Vendored
+164
View File
@@ -0,0 +1,164 @@
use crate::util::{
prefilter::PrefilterI,
search::{MatchKind, Span},
};
#[derive(Clone, Debug)]
pub(crate) struct Teddy {
#[cfg(not(feature = "perf-literal-multisubstring"))]
_unused: (),
/// The actual Teddy searcher.
///
/// Technically, it's possible that Teddy doesn't actually get used, since
/// Teddy does require its haystack to at least be of a certain size
/// (usually around the size of whatever vector is being used, so ~16
/// or ~32 bytes). For haystacks shorter than that, the implementation
/// currently uses Rabin-Karp.
#[cfg(feature = "perf-literal-multisubstring")]
searcher: aho_corasick::packed::Searcher,
/// When running an anchored search, the packed searcher can't handle it so
/// we defer to Aho-Corasick itself. Kind of sad, but changing the packed
/// searchers to support anchored search would be difficult at worst and
/// annoying at best. Since packed searchers only apply to small numbers of
/// literals, we content ourselves that this is not much of an added cost.
/// (That packed searchers only work with a small number of literals is
/// also why we use a DFA here. Otherwise, the memory usage of a DFA would
/// likely be unacceptable.)
#[cfg(feature = "perf-literal-multisubstring")]
anchored_ac: aho_corasick::dfa::DFA,
/// The length of the smallest literal we look for.
///
/// We use this as a heuristic to figure out whether this will be "fast" or
/// not. Generally, the longer the better, because longer needles are more
/// discriminating and thus reduce false positive rate.
#[cfg(feature = "perf-literal-multisubstring")]
minimum_len: usize,
}
impl Teddy {
pub(crate) fn new<B: AsRef<[u8]>>(
kind: MatchKind,
needles: &[B],
) -> Option<Teddy> {
#[cfg(not(feature = "perf-literal-multisubstring"))]
{
None
}
#[cfg(feature = "perf-literal-multisubstring")]
{
// We only really support leftmost-first semantics. In
// theory we could at least support leftmost-longest, as the
// aho-corasick crate does, but regex-automata doesn't know about
// leftmost-longest currently.
//
// And like the aho-corasick prefilter, if we're using `All`
// semantics, then we can still use leftmost semantics for a
// prefilter. (This might be a suspicious choice for the literal
// engine, which uses a prefilter as a regex engine directly, but
// that only happens when using leftmost-first semantics.)
let (packed_match_kind, ac_match_kind) = match kind {
MatchKind::LeftmostFirst | MatchKind::All => (
aho_corasick::packed::MatchKind::LeftmostFirst,
aho_corasick::MatchKind::LeftmostFirst,
),
};
let minimum_len =
needles.iter().map(|n| n.as_ref().len()).min().unwrap_or(0);
let packed = aho_corasick::packed::Config::new()
.match_kind(packed_match_kind)
.builder()
.extend(needles)
.build()?;
let anchored_ac = aho_corasick::dfa::DFA::builder()
.match_kind(ac_match_kind)
.start_kind(aho_corasick::StartKind::Anchored)
.prefilter(false)
.build(needles)
.ok()?;
Some(Teddy { searcher: packed, anchored_ac, minimum_len })
}
}
}
impl PrefilterI for Teddy {
fn name(&self) -> &'static str {
"teddy"
}
fn find(&self, haystack: &[u8], span: Span) -> Option<Span> {
#[cfg(not(feature = "perf-literal-multisubstring"))]
{
unreachable!()
}
#[cfg(feature = "perf-literal-multisubstring")]
{
let ac_span =
aho_corasick::Span { start: span.start, end: span.end };
self.searcher
.find_in(haystack, ac_span)
.map(|m| Span { start: m.start(), end: m.end() })
}
}
fn prefix(&self, haystack: &[u8], span: Span) -> Option<Span> {
#[cfg(not(feature = "perf-literal-multisubstring"))]
{
unreachable!()
}
#[cfg(feature = "perf-literal-multisubstring")]
{
use aho_corasick::automaton::Automaton;
let input = aho_corasick::Input::new(haystack)
.anchored(aho_corasick::Anchored::Yes)
.span(span.start..span.end);
self.anchored_ac
.try_find(&input)
// OK because we build the DFA with anchored support.
.expect("aho-corasick DFA should never fail")
.map(|m| Span { start: m.start(), end: m.end() })
}
}
fn memory_usage(&self) -> usize {
#[cfg(not(feature = "perf-literal-multisubstring"))]
{
unreachable!()
}
#[cfg(feature = "perf-literal-multisubstring")]
{
use aho_corasick::automaton::Automaton;
self.searcher.memory_usage() + self.anchored_ac.memory_usage()
}
}
fn is_fast(&self) -> bool {
#[cfg(not(feature = "perf-literal-multisubstring"))]
{
unreachable!()
}
#[cfg(feature = "perf-literal-multisubstring")]
{
// Teddy is usually quite fast, but I have seen some cases where
// a large number of literals can overwhelm it and make it not so
// fast. We make an educated but conservative guess at a limit, at
// which point, we're not so comfortable thinking Teddy is "fast."
//
// Well... this used to incorporate a "limit" on the *number*
// of literals, but I have since changed it to a minimum on the
// *smallest* literal. Namely, when there is a very small literal
// (1 or 2 bytes), it is far more likely that it leads to a higher
// false positive rate. (Although, of course, not always. For
// example, 'zq' is likely to have a very low false positive rate.)
// But when we have 3 bytes, we have a really good chance of being
// quite discriminatory and thus fast.
//
// We may still want to add some kind of limit on the number of
// literals here, but keep in mind that Teddy already has its own
// somewhat small limit (64 at time of writing). The main issue
// here is that if 'is_fast' is false, it opens the door for the
// reverse inner optimization to kick in. We really only want to
// resort to the reverse inner optimization if we absolutely must.
self.minimum_len >= 3
}
}
}
third_party/crates/regex-automata-0.4.18/src/util/primitives.rs
Vendored
+776
View File
@@ -0,0 +1,776 @@
/*!
Lower level primitive types that are useful in a variety of circumstances.
# Overview
This list represents the principle types in this module and briefly describes
when you might want to use them.
* [`PatternID`] - A type that represents the identifier of a regex pattern.
This is probably the most widely used type in this module (which is why it's
also re-exported in the crate root).
* [`StateID`] - A type the represents the identifier of a finite automaton
state. This is used for both NFAs and DFAs, with the notable exception of
the hybrid NFA/DFA. (The hybrid NFA/DFA uses a special purpose "lazy" state
identifier.)
* [`SmallIndex`] - The internal representation of both a `PatternID` and a
`StateID`. Its purpose is to serve as a type that can index memory without
being as big as a `usize` on 64-bit targets. The main idea behind this type
is that there are many things in regex engines that will, in practice, never
overflow a 32-bit integer. (For example, like the number of patterns in a regex
or the number of states in an NFA.) Thus, a `SmallIndex` can be used to index
memory without peppering `as` casts everywhere. Moreover, it forces callers
to handle errors in the case where, somehow, the value would otherwise overflow
either a 32-bit integer or a `usize` (e.g., on 16-bit targets).
* [`NonMaxUsize`] - Represents a `usize` that cannot be `usize::MAX`. As a
result, `Option<NonMaxUsize>` has the same size in memory as a `usize`. This
useful, for example, when representing the offsets of submatches since it
reduces memory usage by a factor of 2. It is a legal optimization since Rust
guarantees that slices never have a length that exceeds `isize::MAX`.
*/
use core::num::NonZeroUsize;
#[cfg(feature = "alloc")]
use alloc::vec::Vec;
use crate::util::int::{Usize, U16, U32, U64};
/// A `usize` that can never be `usize::MAX`.
///
/// This is similar to `core::num::NonZeroUsize`, but instead of not permitting
/// a zero value, this does not permit a max value.
///
/// This is useful in certain contexts where one wants to optimize the memory
/// usage of things that contain match offsets. Namely, since Rust slices
/// are guaranteed to never have a length exceeding `isize::MAX`, we can use
/// `usize::MAX` as a sentinel to indicate that no match was found. Indeed,
/// types like `Option<NonMaxUsize>` have exactly the same size in memory as a
/// `usize`.
///
/// This type is defined to be `repr(transparent)` for
/// `core::num::NonZeroUsize`, which is in turn defined to be
/// `repr(transparent)` for `usize`.
#[derive(Clone, Copy, Eq, Hash, PartialEq, PartialOrd, Ord)]
#[repr(transparent)]
pub struct NonMaxUsize(NonZeroUsize);
impl NonMaxUsize {
/// Create a new `NonMaxUsize` from the given value.
///
/// This returns `None` only when the given value is equal to `usize::MAX`.
#[inline]
pub fn new(value: usize) -> Option<NonMaxUsize> {
NonZeroUsize::new(value.wrapping_add(1)).map(NonMaxUsize)
}
/// Return the underlying `usize` value. The returned value is guaranteed
/// to not equal `usize::MAX`.
#[inline]
pub fn get(self) -> usize {
self.0.get().wrapping_sub(1)
}
}
// We provide our own Debug impl because seeing the internal repr can be quite
// surprising if you aren't expecting it. e.g., 'NonMaxUsize(5)' vs just '5'.
impl core::fmt::Debug for NonMaxUsize {
fn fmt(&self, f: &mut core::fmt::Formatter) -> core::fmt::Result {
write!(f, "{:?}", self.get())
}
}
/// A type that represents a "small" index.
///
/// The main idea of this type is to provide something that can index memory,
/// but uses less memory than `usize` on 64-bit systems. Specifically, its
/// representation is always a `u32` and has `repr(transparent)` enabled. (So
/// it is safe to transmute between a `u32` and a `SmallIndex`.)
///
/// A small index is typically useful in cases where there is no practical way
/// that the index will overflow a 32-bit integer. A good example of this is
/// an NFA state. If you could somehow build an NFA with `2^30` states, its
/// memory usage would be exorbitant and its runtime execution would be so
/// slow as to be completely worthless. Therefore, this crate generally deems
/// it acceptable to return an error if it would otherwise build an NFA that
/// requires a slice longer than what a 32-bit integer can index. In exchange,
/// we can use 32-bit indices instead of 64-bit indices in various places.
///
/// This type ensures this by providing a constructor that will return an error
/// if its argument cannot fit into the type. This makes it much easier to
/// handle these sorts of boundary cases that are otherwise extremely subtle.
///
/// On all targets, this type guarantees that its value will fit in a `u32`,
/// `i32`, `usize` and an `isize`. This means that on 16-bit targets, for
/// example, this type's maximum value will never overflow an `isize`,
/// which means it will never overflow a `i16` even though its internal
/// representation is still a `u32`.
///
/// The purpose for making the type fit into even signed integer types like
/// `isize` is to guarantee that the difference between any two small indices
/// is itself also a small index. This is useful in certain contexts, e.g.,
/// for delta encoding.
///
/// # Other types
///
/// The following types wrap `SmallIndex` to provide a more focused use case:
///
/// * [`PatternID`] is for representing the identifiers of patterns.
/// * [`StateID`] is for representing the identifiers of states in finite
/// automata. It is used for both NFAs and DFAs.
///
/// # Representation
///
/// This type is always represented internally by a `u32` and is marked as
/// `repr(transparent)`. Thus, this type always has the same representation as
/// a `u32`. It is thus safe to transmute between a `u32` and a `SmallIndex`.
///
/// # Indexing
///
/// For convenience, callers may use a `SmallIndex` to index slices.
///
/// # Safety
///
/// While a `SmallIndex` is meant to guarantee that its value fits into `usize`
/// without using as much space as a `usize` on all targets, callers must
/// not rely on this property for safety. Callers may choose to rely on this
/// property for correctness however. For example, creating a `SmallIndex` with
/// an invalid value can be done in entirely safe code. This may in turn result
/// in panics or silent logical errors.
#[derive(
Clone, Copy, Debug, Default, Eq, Hash, PartialEq, PartialOrd, Ord,
)]
#[repr(transparent)]
pub struct SmallIndex(u32);
impl SmallIndex {
/// The maximum index value.
#[cfg(any(target_pointer_width = "32", target_pointer_width = "64"))]
pub const MAX: SmallIndex =
// FIXME: Use as_usize() once const functions in traits are stable.
SmallIndex::new_unchecked(core::i32::MAX as usize - 1);
/// The maximum index value.
#[cfg(target_pointer_width = "16")]
pub const MAX: SmallIndex =
SmallIndex::new_unchecked(core::isize::MAX - 1);
/// The total number of values that can be represented as a small index.
pub const LIMIT: usize = SmallIndex::MAX.as_usize() + 1;
/// The zero index value.
pub const ZERO: SmallIndex = SmallIndex::new_unchecked(0);
/// The number of bytes that a single small index uses in memory.
pub const SIZE: usize = core::mem::size_of::<SmallIndex>();
/// Create a new small index.
///
/// If the given index exceeds [`SmallIndex::MAX`], then this returns
/// an error.
#[inline]
pub fn new(index: usize) -> Result<SmallIndex, SmallIndexError> {
SmallIndex::try_from(index)
}
/// Create a new small index without checking whether the given value
/// exceeds [`SmallIndex::MAX`].
///
/// Using this routine with an invalid index value will result in
/// unspecified behavior, but *not* undefined behavior. In particular, an
/// invalid index value is likely to cause panics or possibly even silent
/// logical errors.
///
/// Callers must never rely on a `SmallIndex` to be within a certain range
/// for memory safety.
#[inline]
pub const fn new_unchecked(index: usize) -> SmallIndex {
// FIXME: Use as_u32() once const functions in traits are stable.
SmallIndex(index as u32)
}
/// Like [`SmallIndex::new`], but panics if the given index is not valid.
#[inline]
pub fn must(index: usize) -> SmallIndex {
SmallIndex::new(index).expect("invalid small index")
}
/// Return this small index as a `usize`. This is guaranteed to never
/// overflow `usize`.
#[inline]
pub const fn as_usize(&self) -> usize {
// FIXME: Use as_usize() once const functions in traits are stable.
self.0 as usize
}
/// Return this small index as a `u64`. This is guaranteed to never
/// overflow.
#[inline]
pub const fn as_u64(&self) -> u64 {
// FIXME: Use u64::from() once const functions in traits are stable.
self.0 as u64
}
/// Return the internal `u32` of this small index. This is guaranteed to
/// never overflow `u32`.
#[inline]
pub const fn as_u32(&self) -> u32 {
self.0
}
/// Return the internal `u32` of this small index represented as an `i32`.
/// This is guaranteed to never overflow an `i32`.
#[inline]
pub const fn as_i32(&self) -> i32 {
// This is OK because we guarantee that our max value is <= i32::MAX.
self.0 as i32
}
/// Returns one more than this small index as a usize.
///
/// Since a small index has constraints on its maximum value, adding `1` to
/// it will always fit in a `usize`, `u32` and a `i32`.
#[inline]
pub fn one_more(&self) -> usize {
self.as_usize() + 1
}
/// Decode this small index from the bytes given using the native endian
/// byte order for the current target.
///
/// If the decoded integer is not representable as a small index for the
/// current target, then this returns an error.
#[inline]
pub fn from_ne_bytes(
bytes: [u8; 4],
) -> Result<SmallIndex, SmallIndexError> {
let id = u32::from_ne_bytes(bytes);
if id > SmallIndex::MAX.as_u32() {
return Err(SmallIndexError { attempted: u64::from(id) });
}
Ok(SmallIndex::new_unchecked(id.as_usize()))
}
/// Decode this small index from the bytes given using the native endian
/// byte order for the current target.
///
/// This is analogous to [`SmallIndex::new_unchecked`] in that is does not
/// check whether the decoded integer is representable as a small index.
#[inline]
pub fn from_ne_bytes_unchecked(bytes: [u8; 4]) -> SmallIndex {
SmallIndex::new_unchecked(u32::from_ne_bytes(bytes).as_usize())
}
/// Return the underlying small index integer as raw bytes in native endian
/// format.
#[inline]
pub fn to_ne_bytes(&self) -> [u8; 4] {
self.0.to_ne_bytes()
}
}
impl<T> core::ops::Index<SmallIndex> for [T] {
type Output = T;
#[inline]
fn index(&self, index: SmallIndex) -> &T {
&self[index.as_usize()]
}
}
impl<T> core::ops::IndexMut<SmallIndex> for [T] {
#[inline]
fn index_mut(&mut self, index: SmallIndex) -> &mut T {
&mut self[index.as_usize()]
}
}
#[cfg(feature = "alloc")]
impl<T> core::ops::Index<SmallIndex> for Vec<T> {
type Output = T;
#[inline]
fn index(&self, index: SmallIndex) -> &T {
&self[index.as_usize()]
}
}
#[cfg(feature = "alloc")]
impl<T> core::ops::IndexMut<SmallIndex> for Vec<T> {
#[inline]
fn index_mut(&mut self, index: SmallIndex) -> &mut T {
&mut self[index.as_usize()]
}
}
impl From<u8> for SmallIndex {
fn from(index: u8) -> SmallIndex {
SmallIndex::new_unchecked(usize::from(index))
}
}
impl TryFrom<u16> for SmallIndex {
type Error = SmallIndexError;
fn try_from(index: u16) -> Result<SmallIndex, SmallIndexError> {
if u32::from(index) > SmallIndex::MAX.as_u32() {
return Err(SmallIndexError { attempted: u64::from(index) });
}
Ok(SmallIndex::new_unchecked(index.as_usize()))
}
}
impl TryFrom<u32> for SmallIndex {
type Error = SmallIndexError;
fn try_from(index: u32) -> Result<SmallIndex, SmallIndexError> {
if index > SmallIndex::MAX.as_u32() {
return Err(SmallIndexError { attempted: u64::from(index) });
}
Ok(SmallIndex::new_unchecked(index.as_usize()))
}
}
impl TryFrom<u64> for SmallIndex {
type Error = SmallIndexError;
fn try_from(index: u64) -> Result<SmallIndex, SmallIndexError> {
if index > SmallIndex::MAX.as_u64() {
return Err(SmallIndexError { attempted: index });
}
Ok(SmallIndex::new_unchecked(index.as_usize()))
}
}
impl TryFrom<usize> for SmallIndex {
type Error = SmallIndexError;
fn try_from(index: usize) -> Result<SmallIndex, SmallIndexError> {
if index > SmallIndex::MAX.as_usize() {
return Err(SmallIndexError { attempted: index.as_u64() });
}
Ok(SmallIndex::new_unchecked(index))
}
}
#[cfg(test)]
impl quickcheck::Arbitrary for SmallIndex {
fn arbitrary(gen: &mut quickcheck::Gen) -> SmallIndex {
use core::cmp::max;
let id = max(i32::MIN + 1, i32::arbitrary(gen)).abs();
if id > SmallIndex::MAX.as_i32() {
SmallIndex::MAX
} else {
SmallIndex::new(usize::try_from(id).unwrap()).unwrap()
}
}
}
/// This error occurs when a small index could not be constructed.
///
/// This occurs when given an integer exceeding the maximum small index value.
///
/// When the `std` feature is enabled, this implements the `Error` trait.
#[derive(Clone, Debug, Eq, PartialEq)]
pub struct SmallIndexError {
attempted: u64,
}
impl SmallIndexError {
/// Returns the value that could not be converted to a small index.
pub fn attempted(&self) -> u64 {
self.attempted
}
}
#[cfg(feature = "std")]
impl std::error::Error for SmallIndexError {}
impl core::fmt::Display for SmallIndexError {
fn fmt(&self, f: &mut core::fmt::Formatter) -> core::fmt::Result {
write!(
f,
"failed to create small index from {:?}, which exceeds {:?}",
self.attempted(),
SmallIndex::MAX,
)
}
}
#[derive(Clone, Debug)]
pub(crate) struct SmallIndexIter {
rng: core::ops::Range<usize>,
}
impl Iterator for SmallIndexIter {
type Item = SmallIndex;
fn next(&mut self) -> Option<SmallIndex> {
if self.rng.start >= self.rng.end {
return None;
}
let next_id = self.rng.start + 1;
let id = core::mem::replace(&mut self.rng.start, next_id);
// new_unchecked is OK since we asserted that the number of
// elements in this iterator will fit in an ID at construction.
Some(SmallIndex::new_unchecked(id))
}
}
macro_rules! index_type_impls {
($name:ident, $err:ident, $iter:ident, $withiter:ident) => {
impl $name {
/// The maximum value.
pub const MAX: $name = $name(SmallIndex::MAX);
/// The total number of values that can be represented.
pub const LIMIT: usize = SmallIndex::LIMIT;
/// The zero value.
pub const ZERO: $name = $name(SmallIndex::ZERO);
/// The number of bytes that a single value uses in memory.
pub const SIZE: usize = SmallIndex::SIZE;
/// Create a new value that is represented by a "small index."
///
/// If the given index exceeds the maximum allowed value, then this
/// returns an error.
#[inline]
pub fn new(value: usize) -> Result<$name, $err> {
SmallIndex::new(value).map($name).map_err($err)
}
/// Create a new value without checking whether the given argument
/// exceeds the maximum.
///
/// Using this routine with an invalid value will result in
/// unspecified behavior, but *not* undefined behavior. In
/// particular, an invalid ID value is likely to cause panics or
/// possibly even silent logical errors.
///
/// Callers must never rely on this type to be within a certain
/// range for memory safety.
#[inline]
pub const fn new_unchecked(value: usize) -> $name {
$name(SmallIndex::new_unchecked(value))
}
/// Like `new`, but panics if the given value is not valid.
#[inline]
pub fn must(value: usize) -> $name {
$name::new(value).expect(concat!(
"invalid ",
stringify!($name),
" value"
))
}
/// Return the internal value as a `usize`. This is guaranteed to
/// never overflow `usize`.
#[inline]
pub const fn as_usize(&self) -> usize {
self.0.as_usize()
}
/// Return the internal value as a `u64`. This is guaranteed to
/// never overflow.
#[inline]
pub const fn as_u64(&self) -> u64 {
self.0.as_u64()
}
/// Return the internal value as a `u32`. This is guaranteed to
/// never overflow `u32`.
#[inline]
pub const fn as_u32(&self) -> u32 {
self.0.as_u32()
}
/// Return the internal value as a i32`. This is guaranteed to
/// never overflow an `i32`.
#[inline]
pub const fn as_i32(&self) -> i32 {
self.0.as_i32()
}
/// Returns one more than this value as a usize.
///
/// Since values represented by a "small index" have constraints
/// on their maximum value, adding `1` to it will always fit in a
/// `usize`, `u32` and a `i32`.
#[inline]
pub fn one_more(&self) -> usize {
self.0.one_more()
}
/// Decode this value from the bytes given using the native endian
/// byte order for the current target.
///
/// If the decoded integer is not representable as a small index
/// for the current target, then this returns an error.
#[inline]
pub fn from_ne_bytes(bytes: [u8; 4]) -> Result<$name, $err> {
SmallIndex::from_ne_bytes(bytes).map($name).map_err($err)
}
/// Decode this value from the bytes given using the native endian
/// byte order for the current target.
///
/// This is analogous to `new_unchecked` in that is does not check
/// whether the decoded integer is representable as a small index.
#[inline]
pub fn from_ne_bytes_unchecked(bytes: [u8; 4]) -> $name {
$name(SmallIndex::from_ne_bytes_unchecked(bytes))
}
/// Return the underlying integer as raw bytes in native endian
/// format.
#[inline]
pub fn to_ne_bytes(&self) -> [u8; 4] {
self.0.to_ne_bytes()
}
/// Returns an iterator over all values from 0 up to and not
/// including the given length.
///
/// If the given length exceeds this type's limit, then this
/// panics.
pub(crate) fn iter(len: usize) -> $iter {
$iter::new(len)
}
}
// We write our own Debug impl so that we get things like PatternID(5)
// instead of PatternID(SmallIndex(5)).
impl core::fmt::Debug for $name {
fn fmt(&self, f: &mut core::fmt::Formatter) -> core::fmt::Result {
f.debug_tuple(stringify!($name)).field(&self.as_u32()).finish()
}
}
impl<T> core::ops::Index<$name> for [T] {
type Output = T;
#[inline]
fn index(&self, index: $name) -> &T {
&self[index.as_usize()]
}
}
impl<T> core::ops::IndexMut<$name> for [T] {
#[inline]
fn index_mut(&mut self, index: $name) -> &mut T {
&mut self[index.as_usize()]
}
}
#[cfg(feature = "alloc")]
impl<T> core::ops::Index<$name> for Vec<T> {
type Output = T;
#[inline]
fn index(&self, index: $name) -> &T {
&self[index.as_usize()]
}
}
#[cfg(feature = "alloc")]
impl<T> core::ops::IndexMut<$name> for Vec<T> {
#[inline]
fn index_mut(&mut self, index: $name) -> &mut T {
&mut self[index.as_usize()]
}
}
impl From<u8> for $name {
fn from(value: u8) -> $name {
$name(SmallIndex::from(value))
}
}
impl TryFrom<u16> for $name {
type Error = $err;
fn try_from(value: u16) -> Result<$name, $err> {
SmallIndex::try_from(value).map($name).map_err($err)
}
}
impl TryFrom<u32> for $name {
type Error = $err;
fn try_from(value: u32) -> Result<$name, $err> {
SmallIndex::try_from(value).map($name).map_err($err)
}
}
impl TryFrom<u64> for $name {
type Error = $err;
fn try_from(value: u64) -> Result<$name, $err> {
SmallIndex::try_from(value).map($name).map_err($err)
}
}
impl TryFrom<usize> for $name {
type Error = $err;
fn try_from(value: usize) -> Result<$name, $err> {
SmallIndex::try_from(value).map($name).map_err($err)
}
}
#[cfg(test)]
impl quickcheck::Arbitrary for $name {
fn arbitrary(gen: &mut quickcheck::Gen) -> $name {
$name(SmallIndex::arbitrary(gen))
}
}
/// This error occurs when a value could not be constructed.
///
/// This occurs when given an integer exceeding the maximum allowed
/// value.
///
/// When the `std` feature is enabled, this implements the `Error`
/// trait.
#[derive(Clone, Debug, Eq, PartialEq)]
pub struct $err(SmallIndexError);
impl $err {
/// Returns the value that could not be converted to an ID.
pub fn attempted(&self) -> u64 {
self.0.attempted()
}
}
#[cfg(feature = "std")]
impl std::error::Error for $err {}
impl core::fmt::Display for $err {
fn fmt(&self, f: &mut core::fmt::Formatter) -> core::fmt::Result {
write!(
f,
"failed to create {} from {:?}, which exceeds {:?}",
stringify!($name),
self.attempted(),
$name::MAX,
)
}
}
#[derive(Clone, Debug)]
pub(crate) struct $iter(SmallIndexIter);
impl $iter {
fn new(len: usize) -> $iter {
assert!(
len <= $name::LIMIT,
"cannot create iterator for {} when number of \
elements exceed {:?}",
stringify!($name),
$name::LIMIT,
);
$iter(SmallIndexIter { rng: 0..len })
}
}
impl Iterator for $iter {
type Item = $name;
fn next(&mut self) -> Option<$name> {
self.0.next().map($name)
}
}
/// An iterator adapter that is like std::iter::Enumerate, but attaches
/// small index values instead. It requires `ExactSizeIterator`. At
/// construction, it ensures that the index of each element in the
/// iterator is representable in the corresponding small index type.
#[derive(Clone, Debug)]
pub(crate) struct $withiter<I> {
it: I,
ids: $iter,
}
impl<I: Iterator + ExactSizeIterator> $withiter<I> {
fn new(it: I) -> $withiter<I> {
let ids = $name::iter(it.len());
$withiter { it, ids }
}
}
impl<I: Iterator + ExactSizeIterator> Iterator for $withiter<I> {
type Item = ($name, I::Item);
fn next(&mut self) -> Option<($name, I::Item)> {
let item = self.it.next()?;
// Number of elements in this iterator must match, according
// to contract of ExactSizeIterator.
let id = self.ids.next().unwrap();
Some((id, item))
}
}
};
}
/// The identifier of a regex pattern, represented by a [`SmallIndex`].
///
/// The identifier for a pattern corresponds to its relative position among
/// other patterns in a single finite state machine. Namely, when building
/// a multi-pattern regex engine, one must supply a sequence of patterns to
/// match. The position (starting at 0) of each pattern in that sequence
/// represents its identifier. This identifier is in turn used to identify and
/// report matches of that pattern in various APIs.
///
/// See the [`SmallIndex`] type for more information about what it means for
/// a pattern ID to be a "small index."
///
/// Note that this type is defined in the
/// [`util::primitives`](crate::util::primitives) module, but it is also
/// re-exported at the crate root due to how common it is.
#[derive(Clone, Copy, Default, Eq, Hash, PartialEq, PartialOrd, Ord)]
#[repr(transparent)]
pub struct PatternID(SmallIndex);
/// The identifier of a finite automaton state, represented by a
/// [`SmallIndex`].
///
/// Most regex engines in this crate are built on top of finite automata. Each
/// state in a finite automaton defines transitions from its state to another.
/// Those transitions point to other states via their identifiers, i.e., a
/// `StateID`. Since finite automata tend to contain many transitions, it is
/// much more memory efficient to define state IDs as small indices.
///
/// See the [`SmallIndex`] type for more information about what it means for
/// a state ID to be a "small index."
#[derive(Clone, Copy, Default, Eq, Hash, PartialEq, PartialOrd, Ord)]
#[repr(transparent)]
pub struct StateID(SmallIndex);
index_type_impls!(PatternID, PatternIDError, PatternIDIter, WithPatternIDIter);
index_type_impls!(StateID, StateIDError, StateIDIter, WithStateIDIter);
/// A utility trait that defines a couple of adapters for making it convenient
/// to access indices as "small index" types. We require ExactSizeIterator so
/// that iterator construction can do a single check to make sure the index of
/// each element is representable by its small index type.
pub(crate) trait IteratorIndexExt: Iterator {
fn with_pattern_ids(self) -> WithPatternIDIter<Self>
where
Self: Sized + ExactSizeIterator,
{
WithPatternIDIter::new(self)
}
fn with_state_ids(self) -> WithStateIDIter<Self>
where
Self: Sized + ExactSizeIterator,
{
WithStateIDIter::new(self)
}
}
impl<I: Iterator> IteratorIndexExt for I {}
third_party/crates/regex-automata-0.4.18/src/util/search.rs
Vendored
+1997
View File
File diff suppressed because it is too large. Load diff
third_party/crates/regex-automata-0.4.18/src/util/sparse_set.rs
Vendored
+239
View File
@@ -0,0 +1,239 @@
/*!
This module defines a sparse set data structure. Its most interesting
properties are:
* They preserve insertion order.
* Set membership testing is done in constant time.
* Set insertion is done in constant time.
* Clearing the set is done in constant time.
The cost for doing this is that the capacity of the set needs to be known up
front, and the elements in the set are limited to state identifiers.
These sets are principally used when traversing an NFA state graph. This
happens at search time, for example, in the PikeVM. It also happens during DFA
determinization.
*/
use alloc::{vec, vec::Vec};
use crate::util::primitives::StateID;
/// A pair of sparse sets.
///
/// This is useful when one needs to compute NFA epsilon closures from a
/// previous set of states derived from an epsilon closure. One set can be the
/// starting states where as the other set can be the destination states after
/// following the transitions for a particular byte of input.
///
/// There is no significance to 'set1' or 'set2'. They are both sparse sets of
/// the same size.
///
/// The members of this struct are exposed so that callers may borrow 'set1'
/// and 'set2' individually without being force to borrow both at the same
/// time.
#[derive(Clone, Debug)]
pub(crate) struct SparseSets {
pub(crate) set1: SparseSet,
pub(crate) set2: SparseSet,
}
impl SparseSets {
/// Create a new pair of sparse sets where each set has the given capacity.
///
/// This panics if the capacity given is bigger than `StateID::LIMIT`.
pub(crate) fn new(capacity: usize) -> SparseSets {
SparseSets {
set1: SparseSet::new(capacity),
set2: SparseSet::new(capacity),
}
}
/// Resizes these sparse sets to have the new capacity given.
///
/// The sets are automatically cleared.
///
/// This panics if the capacity given is bigger than `StateID::LIMIT`.
#[inline]
pub(crate) fn resize(&mut self, new_capacity: usize) {
self.set1.resize(new_capacity);
self.set2.resize(new_capacity);
}
/// Clear both sparse sets.
pub(crate) fn clear(&mut self) {
self.set1.clear();
self.set2.clear();
}
/// Swap set1 with set2.
pub(crate) fn swap(&mut self) {
core::mem::swap(&mut self.set1, &mut self.set2);
}
/// Returns the memory usage, in bytes, used by this pair of sparse sets.
pub(crate) fn memory_usage(&self) -> usize {
self.set1.memory_usage() + self.set2.memory_usage()
}
}
/// A sparse set used for representing ordered NFA states.
///
/// This supports constant time addition and membership testing. Clearing an
/// entire set can also be done in constant time. Iteration yields elements
/// in the order in which they were inserted.
///
/// The data structure is based on: https://research.swtch.com/sparse
/// Note though that we don't actually use uninitialized memory. We generally
/// reuse sparse sets, so the initial allocation cost is bearable. However, its
/// other properties listed above are extremely useful.
#[derive(Clone)]
pub(crate) struct SparseSet {
/// The number of elements currently in this set.
len: usize,
/// Dense contains the ids in the order in which they were inserted.
dense: Vec<StateID>,
/// Sparse maps ids to their location in dense.
///
/// A state ID is in the set if and only if
/// sparse[id] < len && id == dense[sparse[id]].
///
/// Note that these are indices into 'dense'. It's a little weird to use
/// StateID here, but we know our length can never exceed the bounds of
/// StateID (enforced by 'resize') and StateID will be at most 4 bytes
/// where as a usize is likely double that in most cases.
sparse: Vec<StateID>,
}
impl SparseSet {
/// Create a new sparse set with the given capacity.
///
/// Sparse sets have a fixed size and they cannot grow. Attempting to
/// insert more distinct elements than the total capacity of the set will
/// result in a panic.
///
/// This panics if the capacity given is bigger than `StateID::LIMIT`.
#[inline]
pub(crate) fn new(capacity: usize) -> SparseSet {
let mut set = SparseSet { len: 0, dense: vec![], sparse: vec![] };
set.resize(capacity);
set
}
/// Resizes this sparse set to have the new capacity given.
///
/// This set is automatically cleared.
///
/// This panics if the capacity given is bigger than `StateID::LIMIT`.
#[inline]
pub(crate) fn resize(&mut self, new_capacity: usize) {
assert!(
new_capacity <= StateID::LIMIT,
"sparse set capacity cannot exceed {:?}",
StateID::LIMIT
);
self.clear();
self.dense.resize(new_capacity, StateID::ZERO);
self.sparse.resize(new_capacity, StateID::ZERO);
}
/// Returns the capacity of this set.
///
/// The capacity represents a fixed limit on the number of distinct
/// elements that are allowed in this set. The capacity cannot be changed.
#[inline]
pub(crate) fn capacity(&self) -> usize {
self.dense.len()
}
/// Returns the number of elements in this set.
#[inline]
pub(crate) fn len(&self) -> usize {
self.len
}
/// Returns true if and only if this set is empty.
#[inline]
pub(crate) fn is_empty(&self) -> bool {
self.len() == 0
}
/// Insert the state ID value into this set and return true if the given
/// state ID was not previously in this set.
///
/// This operation is idempotent. If the given value is already in this
/// set, then this is a no-op.
///
/// If more than `capacity` ids are inserted, then this panics.
///
/// This is marked as inline(always) since the compiler won't inline it
/// otherwise, and it's a fairly hot piece of code in DFA determinization.
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn insert(&mut self, id: StateID) -> bool {
if self.contains(id) {
return false;
}
let i = self.len();
assert!(
i < self.capacity(),
"{:?} exceeds capacity of {:?} when inserting {:?}",
i,
self.capacity(),
id,
);
// OK since i < self.capacity() and self.capacity() is guaranteed to
// be <= StateID::LIMIT.
let index = StateID::new_unchecked(i);
self.dense[index] = id;
self.sparse[id] = index;
self.len += 1;
true
}
/// Returns true if and only if this set contains the given value.
#[inline]
pub(crate) fn contains(&self, id: StateID) -> bool {
let index = self.sparse[id];
index.as_usize() < self.len() && self.dense[index] == id
}
/// Clear this set such that it has no members.
#[inline]
pub(crate) fn clear(&mut self) {
self.len = 0;
}
#[inline]
pub(crate) fn iter(&self) -> SparseSetIter<'_> {
SparseSetIter(self.dense[..self.len()].iter())
}
/// Returns the heap memory usage, in bytes, used by this sparse set.
#[inline]
pub(crate) fn memory_usage(&self) -> usize {
self.dense.len() * StateID::SIZE + self.sparse.len() * StateID::SIZE
}
}
impl core::fmt::Debug for SparseSet {
fn fmt(&self, f: &mut core::fmt::Formatter) -> core::fmt::Result {
let elements: Vec<StateID> = self.iter().collect();
f.debug_tuple("SparseSet").field(&elements).finish()
}
}
/// An iterator over all elements in a sparse set.
///
/// The lifetime `'a` refers to the lifetime of the set being iterated over.
#[derive(Debug)]
pub(crate) struct SparseSetIter<'a>(core::slice::Iter<'a, StateID>);
impl<'a> Iterator for SparseSetIter<'a> {
type Item = StateID;
#[cfg_attr(feature = "perf-inline", inline(always))]
fn next(&mut self) -> Option<StateID> {
self.0.next().copied()
}
}
third_party/crates/regex-automata-0.4.18/src/util/start.rs
Vendored
+479
View File
@@ -0,0 +1,479 @@
/*!
Provides helpers for dealing with start state configurations in DFAs.
*/
use crate::util::{
look::LookMatcher,
search::{Anchored, Input},
wire::{self, DeserializeError, SerializeError},
};
/// The configuration used to determine a DFA's start state for a search.
///
/// A DFA has a single starting state in the typical textbook description. That
/// is, it corresponds to the set of all starting states for the NFA that built
/// it, along with their epsilon closures. In this crate, however, DFAs have
/// many possible start states due to a few factors:
///
/// * DFAs support the ability to run either anchored or unanchored searches.
/// Each type of search needs its own start state. For example, an unanchored
/// search requires starting at a state corresponding to a regex with a
/// `(?s-u:.)*?` prefix, which will match through anything.
/// * DFAs also optionally support starting an anchored search for any one
/// specific pattern. Each such pattern requires its own start state.
/// * If a look-behind assertion like `^` or `\b` is used in the regex, then
/// the DFA will need to inspect a single byte immediately before the start of
/// the search to choose the correct start state.
///
/// Indeed, this configuration precisely encapsulates all of the above factors.
/// The [`Config::anchored`] method sets which kind of anchored search to
/// perform while the [`Config::look_behind`] method provides a way to set
/// the byte that occurs immediately before the start of the search.
///
/// Generally speaking, this type is only useful when you want to run searches
/// without using an [`Input`]. In particular, an `Input` wants a haystack
/// slice, but callers may not have a contiguous sequence of bytes as a
/// haystack in all cases. This type provides a lower level of control such
/// that callers can provide their own anchored configuration and look-behind
/// byte explicitly.
///
/// # Example
///
/// This shows basic usage that permits running a search with a DFA without
/// using the `Input` abstraction.
///
/// ```
/// use regex_automata::{
/// dfa::{Automaton, dense},
/// util::start,
/// Anchored,
/// };
///
/// let dfa = dense::DFA::new(r"(?-u)\b\w+\b")?;
/// let haystack = "quartz";
///
/// let config = start::Config::new().anchored(Anchored::Yes);
/// let mut state = dfa.start_state(&config)?;
/// for &b in haystack.as_bytes().iter() {
/// state = dfa.next_state(state, b);
/// }
/// state = dfa.next_eoi_state(state);
/// assert!(dfa.is_match_state(state));
///
/// # Ok::<(), Box<dyn std::error::Error>>(())
/// ```
///
/// This example shows how to correctly run a search that doesn't begin at
/// the start of a haystack. Notice how we set the look-behind byte, and as
/// a result, the `\b` assertion does not match.
///
/// ```
/// use regex_automata::{
/// dfa::{Automaton, dense},
/// util::start,
/// Anchored,
/// };
///
/// let dfa = dense::DFA::new(r"(?-u)\b\w+\b")?;
/// let haystack = "quartz";
///
/// let config = start::Config::new()
/// .anchored(Anchored::Yes)
/// .look_behind(Some(b'q'));
/// let mut state = dfa.start_state(&config)?;
/// for &b in haystack.as_bytes().iter().skip(1) {
/// state = dfa.next_state(state, b);
/// }
/// state = dfa.next_eoi_state(state);
/// // No match!
/// assert!(!dfa.is_match_state(state));
///
/// # Ok::<(), Box<dyn std::error::Error>>(())
/// ```
///
/// If we had instead not set a look-behind byte, then the DFA would assume
/// that it was starting at the beginning of the haystack, and thus `\b` should
/// match. This in turn would result in erroneously reporting a match:
///
/// ```
/// use regex_automata::{
/// dfa::{Automaton, dense},
/// util::start,
/// Anchored,
/// };
///
/// let dfa = dense::DFA::new(r"(?-u)\b\w+\b")?;
/// let haystack = "quartz";
///
/// // Whoops, forgot the look-behind byte...
/// let config = start::Config::new().anchored(Anchored::Yes);
/// let mut state = dfa.start_state(&config)?;
/// for &b in haystack.as_bytes().iter().skip(1) {
/// state = dfa.next_state(state, b);
/// }
/// state = dfa.next_eoi_state(state);
/// // And now we get a match unexpectedly.
/// assert!(dfa.is_match_state(state));
///
/// # Ok::<(), Box<dyn std::error::Error>>(())
/// ```
#[derive(Clone, Debug)]
pub struct Config {
look_behind: Option<u8>,
anchored: Anchored,
}
impl Config {
/// Create a new default start configuration.
///
/// The default is an unanchored search that starts at the beginning of the
/// haystack.
pub fn new() -> Config {
Config { anchored: Anchored::No, look_behind: None }
}
/// A convenience routine for building a start configuration from an
/// [`Input`] for a forward search.
///
/// This automatically sets the look-behind byte to the byte immediately
/// preceding the start of the search. If the start of the search is at
/// offset `0`, then no look-behind byte is set.
pub fn from_input_forward(input: &Input<'_>) -> Config {
let look_behind = input
.start()
.checked_sub(1)
.and_then(|i| input.haystack().get(i).copied());
Config { look_behind, anchored: input.get_anchored() }
}
/// A convenience routine for building a start configuration from an
/// [`Input`] for a reverse search.
///
/// This automatically sets the look-behind byte to the byte immediately
/// following the end of the search. If the end of the search is at
/// offset `haystack.len()`, then no look-behind byte is set.
pub fn from_input_reverse(input: &Input<'_>) -> Config {
let look_behind = input.haystack().get(input.end()).copied();
Config { look_behind, anchored: input.get_anchored() }
}
/// Set the look-behind byte at the start of a search.
///
/// Unless the search is intended to logically start at the beginning of a
/// haystack, this should _always_ be set to the byte immediately preceding
/// the start of the search. If no look-behind byte is set, then the start
/// configuration will assume it is at the beginning of the haystack. For
/// example, the anchor `^` will match.
///
/// The default is that no look-behind byte is set.
pub fn look_behind(mut self, byte: Option<u8>) -> Config {
self.look_behind = byte;
self
}
/// Set the anchored mode of a search.
///
/// The default is an unanchored search.
pub fn anchored(mut self, mode: Anchored) -> Config {
self.anchored = mode;
self
}
/// Return the look-behind byte in this configuration, if one exists.
pub fn get_look_behind(&self) -> Option<u8> {
self.look_behind
}
/// Return the anchored mode in this configuration.
pub fn get_anchored(&self) -> Anchored {
self.anchored
}
}
/// A map from every possible byte value to its corresponding starting
/// configuration.
///
/// This map is used in order to lookup the start configuration for a particular
/// position in a haystack. This start configuration is then used in
/// combination with things like the anchored mode and pattern ID to fully
/// determine the start state.
///
/// Generally speaking, this map is only used for fully compiled DFAs and lazy
/// DFAs. For NFAs (including the one-pass DFA), the start state is generally
/// selected by virtue of traversing the NFA state graph. DFAs do the same
/// thing, but at build time and not search time. (Well, technically the lazy
/// DFA does it at search time, but it does enough work to cache the full
/// result of the epsilon closure that the NFA engines tend to need to do.)
#[derive(Clone)]
pub(crate) struct StartByteMap {
map: [Start; 256],
}
impl StartByteMap {
/// Create a new map from byte values to their corresponding starting
/// configurations. The map is determined, in part, by how look-around
/// assertions are matched via the matcher given.
pub(crate) fn new(lookm: &LookMatcher) -> StartByteMap {
let mut map = [Start::NonWordByte; 256];
map[usize::from(b'\n')] = Start::LineLF;
map[usize::from(b'\r')] = Start::LineCR;
map[usize::from(b'_')] = Start::WordByte;
let mut byte = b'0';
while byte <= b'9' {
map[usize::from(byte)] = Start::WordByte;
byte += 1;
}
byte = b'A';
while byte <= b'Z' {
map[usize::from(byte)] = Start::WordByte;
byte += 1;
}
byte = b'a';
while byte <= b'z' {
map[usize::from(byte)] = Start::WordByte;
byte += 1;
}
let lineterm = lookm.get_line_terminator();
// If our line terminator is normal, then it is already handled by
// the LineLF and LineCR configurations. But if it's weird, then we
// overwrite whatever was there before for that terminator with a
// special configuration. The trick here is that if the terminator
// is, say, a word byte like `a`, then callers seeing this start
// configuration need to account for that and build their DFA state as
// if it *also* came from a word byte.
if lineterm != b'\r' && lineterm != b'\n' {
map[usize::from(lineterm)] = Start::CustomLineTerminator;
}
StartByteMap { map }
}
/// Return the starting configuration for the given look-behind byte.
///
/// If no look-behind exists, callers should use `Start::Text`.
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn get(&self, byte: u8) -> Start {
self.map[usize::from(byte)]
}
/// Deserializes a byte class map from the given slice. If the slice is of
/// insufficient length or otherwise contains an impossible mapping, then
/// an error is returned. Upon success, the number of bytes read along with
/// the map are returned. The number of bytes read is always a multiple of
/// 8.
pub(crate) fn from_bytes(
slice: &[u8],
) -> Result<(StartByteMap, usize), DeserializeError> {
wire::check_slice_len(slice, 256, "start byte map")?;
let mut map = [Start::NonWordByte; 256];
for (i, &repr) in slice[..256].iter().enumerate() {
map[i] = match Start::from_usize(usize::from(repr)) {
Some(start) => start,
None => {
return Err(DeserializeError::generic(
"found invalid starting configuration",
))
}
};
}
Ok((StartByteMap { map }, 256))
}
/// Writes this map to the given byte buffer. if the given buffer is too
/// small, then an error is returned. Upon success, the total number of
/// bytes written is returned. The number of bytes written is guaranteed to
/// be a multiple of 8.
pub(crate) fn write_to(
&self,
dst: &mut [u8],
) -> Result<usize, SerializeError> {
let nwrite = self.write_to_len();
if dst.len() < nwrite {
return Err(SerializeError::buffer_too_small("start byte map"));
}
for (i, &start) in self.map.iter().enumerate() {
dst[i] = start.as_u8();
}
Ok(nwrite)
}
/// Returns the total number of bytes written by `write_to`.
pub(crate) fn write_to_len(&self) -> usize {
256
}
}
impl core::fmt::Debug for StartByteMap {
fn fmt(&self, f: &mut core::fmt::Formatter) -> core::fmt::Result {
use crate::util::escape::DebugByte;
write!(f, "StartByteMap{{")?;
for byte in 0..=255 {
if byte > 0 {
write!(f, ", ")?;
}
let start = self.map[usize::from(byte)];
write!(f, "{:?} => {:?}", DebugByte(byte), start)?;
}
write!(f, "}}")?;
Ok(())
}
}
/// Represents the six possible starting configurations of a DFA search.
///
/// The starting configuration is determined by inspecting the beginning
/// of the haystack (up to 1 byte). Ultimately, this along with a pattern ID
/// (if specified) and the type of search (anchored or not) is what selects the
/// start state to use in a DFA.
///
/// As one example, if a DFA only supports unanchored searches and does not
/// support anchored searches for each pattern, then it will have at most 6
/// distinct start states. (Some start states may be reused if determinization
/// can determine that they will be equivalent.) If the DFA supports both
/// anchored and unanchored searches, then it will have a maximum of 12
/// distinct start states. Finally, if the DFA also supports anchored searches
/// for each pattern, then it can have up to `12 + (N * 6)` start states, where
/// `N` is the number of patterns.
///
/// Handling each of these starting configurations in the context of DFA
/// determinization can be *quite* tricky and subtle. But the code is small
/// and can be found at `crate::util::determinize::set_lookbehind_from_start`.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(crate) enum Start {
/// This occurs when the starting position is not any of the ones below.
NonWordByte = 0,
/// This occurs when the byte immediately preceding the start of the search
/// is an ASCII word byte.
WordByte = 1,
/// This occurs when the starting position of the search corresponds to the
/// beginning of the haystack.
Text = 2,
/// This occurs when the byte immediately preceding the start of the search
/// is a line terminator. Specifically, `\n`.
LineLF = 3,
/// This occurs when the byte immediately preceding the start of the search
/// is a line terminator. Specifically, `\r`.
LineCR = 4,
/// This occurs when a custom line terminator has been set via a
/// `LookMatcher`, and when that line terminator is neither a `\r` or a
/// `\n`.
///
/// If the custom line terminator is a word byte, then this start
/// configuration is still selected. DFAs that implement word boundary
/// assertions will likely need to check whether the custom line terminator
/// is a word byte, in which case, it should behave as if the byte
/// satisfies `\b` in addition to multi-line anchors.
CustomLineTerminator = 5,
}
impl Start {
/// Return the starting state corresponding to the given integer. If no
/// starting state exists for the given integer, then None is returned.
pub(crate) fn from_usize(n: usize) -> Option<Start> {
match n {
0 => Some(Start::NonWordByte),
1 => Some(Start::WordByte),
2 => Some(Start::Text),
3 => Some(Start::LineLF),
4 => Some(Start::LineCR),
5 => Some(Start::CustomLineTerminator),
_ => None,
}
}
/// Returns the total number of starting state configurations.
pub(crate) fn len() -> usize {
6
}
/// Return this starting configuration as `u8` integer. It is guaranteed to
/// be less than `Start::len()`.
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn as_u8(&self) -> u8 {
// AFAIK, 'as' is the only way to zero-cost convert an int enum to an
// actual int.
*self as u8
}
/// Return this starting configuration as a `usize` integer. It is
/// guaranteed to be less than `Start::len()`.
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn as_usize(&self) -> usize {
usize::from(self.as_u8())
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn start_fwd_done_range() {
let smap = StartByteMap::new(&LookMatcher::default());
let input = Input::new("").range(1..0);
let config = Config::from_input_forward(&input);
let start =
config.get_look_behind().map_or(Start::Text, |b| smap.get(b));
assert_eq!(Start::Text, start);
}
#[test]
fn start_rev_done_range() {
let smap = StartByteMap::new(&LookMatcher::default());
let input = Input::new("").range(1..0);
let config = Config::from_input_reverse(&input);
let start =
config.get_look_behind().map_or(Start::Text, |b| smap.get(b));
assert_eq!(Start::Text, start);
}
#[test]
fn start_fwd() {
let f = |haystack, start, end| {
let smap = StartByteMap::new(&LookMatcher::default());
let input = Input::new(haystack).range(start..end);
let config = Config::from_input_forward(&input);
let start =
config.get_look_behind().map_or(Start::Text, |b| smap.get(b));
start
};
assert_eq!(Start::Text, f("", 0, 0));
assert_eq!(Start::Text, f("abc", 0, 3));
assert_eq!(Start::Text, f("\nabc", 0, 3));
assert_eq!(Start::LineLF, f("\nabc", 1, 3));
assert_eq!(Start::LineCR, f("\rabc", 1, 3));
assert_eq!(Start::WordByte, f("abc", 1, 3));
assert_eq!(Start::NonWordByte, f(" abc", 1, 3));
}
#[test]
fn start_rev() {
let f = |haystack, start, end| {
let smap = StartByteMap::new(&LookMatcher::default());
let input = Input::new(haystack).range(start..end);
let config = Config::from_input_reverse(&input);
let start =
config.get_look_behind().map_or(Start::Text, |b| smap.get(b));
start
};
assert_eq!(Start::Text, f("", 0, 0));
assert_eq!(Start::Text, f("abc", 0, 3));
assert_eq!(Start::Text, f("abc\n", 0, 4));
assert_eq!(Start::LineLF, f("abc\nz", 0, 3));
assert_eq!(Start::LineCR, f("abc\rz", 0, 3));
assert_eq!(Start::WordByte, f("abc", 0, 2));
assert_eq!(Start::NonWordByte, f("abc ", 0, 3));
}
}
third_party/crates/regex-automata-0.4.18/src/util/syntax.rs
Vendored
+482
View File
@@ -0,0 +1,482 @@
/*!
Utilities for dealing with the syntax of a regular expression.
This module currently only exposes a [`Config`] type that
itself represents a wrapper around the configuration for a
[`regex-syntax::ParserBuilder`](regex_syntax::ParserBuilder). The purpose of
this wrapper is to make configuring syntax options very similar to how other
configuration is done throughout this crate. Namely, instead of duplicating
syntax options across every builder (of which there are many), we instead
create small config objects like this one that can be passed around and
composed.
*/
use alloc::{vec, vec::Vec};
use regex_syntax::{
ast,
hir::{self, Hir},
Error, ParserBuilder,
};
/// A convenience routine for parsing a pattern into an HIR value with the
/// default configuration.
///
/// # Example
///
/// This shows how to parse a pattern into an HIR value:
///
/// ```
/// use regex_automata::util::syntax;
///
/// let hir = syntax::parse(r"([a-z]+)|([0-9]+)")?;
/// assert_eq!(Some(1), hir.properties().static_explicit_captures_len());
///
/// # Ok::<(), Box<dyn std::error::Error>>(())
/// ```
pub fn parse(pattern: &str) -> Result<Hir, Error> {
parse_with(pattern, &Config::default())
}
/// A convenience routine for parsing many patterns into HIR value with the
/// default configuration.
///
/// # Example
///
/// This shows how to parse many patterns into an corresponding HIR values:
///
/// ```
/// use {
/// regex_automata::util::syntax,
/// regex_syntax::hir::Properties,
/// };
///
/// let hirs = syntax::parse_many(&[
/// r"([a-z]+)|([0-9]+)",
/// r"foo(A-Z]+)bar",
/// ])?;
/// let props = Properties::union(hirs.iter().map(|h| h.properties()));
/// assert_eq!(Some(1), props.static_explicit_captures_len());
///
/// # Ok::<(), Box<dyn std::error::Error>>(())
/// ```
pub fn parse_many<P: AsRef<str>>(patterns: &[P]) -> Result<Vec<Hir>, Error> {
parse_many_with(patterns, &Config::default())
}
/// A convenience routine for parsing a pattern into an HIR value using a
/// `Config`.
///
/// # Example
///
/// This shows how to parse a pattern into an HIR value with a non-default
/// configuration:
///
/// ```
/// use regex_automata::util::syntax;
///
/// let hir = syntax::parse_with(
/// r"^[a-z]+$",
/// &syntax::Config::new().multi_line(true).crlf(true),
/// )?;
/// assert!(hir.properties().look_set().contains_anchor_crlf());
///
/// # Ok::<(), Box<dyn std::error::Error>>(())
/// ```
pub fn parse_with(pattern: &str, config: &Config) -> Result<Hir, Error> {
let mut builder = ParserBuilder::new();
config.apply(&mut builder);
builder.build().parse(pattern)
}
/// A convenience routine for parsing many patterns into HIR values using a
/// `Config`.
///
/// # Example
///
/// This shows how to parse many patterns into an corresponding HIR values
/// with a non-default configuration:
///
/// ```
/// use {
/// regex_automata::util::syntax,
/// regex_syntax::hir::Properties,
/// };
///
/// let patterns = &[
/// r"([a-z]+)|([0-9]+)",
/// r"\W",
/// r"foo(A-Z]+)bar",
/// ];
/// let config = syntax::Config::new().unicode(false).utf8(false);
/// let hirs = syntax::parse_many_with(patterns, &config)?;
/// let props = Properties::union(hirs.iter().map(|h| h.properties()));
/// assert!(!props.is_utf8());
///
/// # Ok::<(), Box<dyn std::error::Error>>(())
/// ```
pub fn parse_many_with<P: AsRef<str>>(
patterns: &[P],
config: &Config,
) -> Result<Vec<Hir>, Error> {
let mut builder = ParserBuilder::new();
config.apply(&mut builder);
let mut hirs = vec![];
for p in patterns.iter() {
hirs.push(builder.build().parse(p.as_ref())?);
}
Ok(hirs)
}
/// A common set of configuration options that apply to the syntax of a regex.
///
/// This represents a group of configuration options that specifically apply
/// to how the concrete syntax of a regular expression is interpreted. In
/// particular, they are generally forwarded to the
/// [`ParserBuilder`](https://docs.rs/regex-syntax/*/regex_syntax/struct.ParserBuilder.html)
/// in the
/// [`regex-syntax`](https://docs.rs/regex-syntax)
/// crate when building a regex from its concrete syntax directly.
///
/// These options are defined as a group since they apply to every regex engine
/// in this crate. Instead of re-defining them on every engine's builder, they
/// are instead provided here as one cohesive unit.
#[derive(Clone, Copy, Debug)]
pub struct Config {
case_insensitive: bool,
multi_line: bool,
dot_matches_new_line: bool,
crlf: bool,
line_terminator: u8,
swap_greed: bool,
ignore_whitespace: bool,
unicode: bool,
utf8: bool,
nest_limit: u32,
octal: bool,
}
impl Config {
/// Return a new default syntax configuration.
pub fn new() -> Config {
// These defaults match the ones used in regex-syntax.
Config {
case_insensitive: false,
multi_line: false,
dot_matches_new_line: false,
crlf: false,
line_terminator: b'\n',
swap_greed: false,
ignore_whitespace: false,
unicode: true,
utf8: true,
nest_limit: 250,
octal: false,
}
}
/// Enable or disable the case insensitive flag by default.
///
/// When Unicode mode is enabled, case insensitivity is Unicode-aware.
/// Specifically, it will apply the "simple" case folding rules as
/// specified by Unicode.
///
/// By default this is disabled. It may alternatively be selectively
/// enabled in the regular expression itself via the `i` flag.
pub fn case_insensitive(mut self, yes: bool) -> Config {
self.case_insensitive = yes;
self
}
/// Enable or disable the multi-line matching flag by default.
///
/// When this is enabled, the `^` and `$` look-around assertions will
/// match immediately after and immediately before a new line character,
/// respectively. Note that the `\A` and `\z` look-around assertions are
/// unaffected by this setting and always correspond to matching at the
/// beginning and end of the input.
///
/// By default this is disabled. It may alternatively be selectively
/// enabled in the regular expression itself via the `m` flag.
pub fn multi_line(mut self, yes: bool) -> Config {
self.multi_line = yes;
self
}
/// Enable or disable the "dot matches any character" flag by default.
///
/// When this is enabled, `.` will match any character. When it's disabled,
/// then `.` will match any character except for a new line character.
///
/// Note that `.` is impacted by whether the "unicode" setting is enabled
/// or not. When Unicode is enabled (the default), `.` will match any UTF-8
/// encoding of any Unicode scalar value (sans a new line, depending on
/// whether this "dot matches new line" option is enabled). When Unicode
/// mode is disabled, `.` will match any byte instead. Because of this,
/// when Unicode mode is disabled, `.` can only be used when the "allow
/// invalid UTF-8" option is enabled, since `.` could otherwise match
/// invalid UTF-8.
///
/// By default this is disabled. It may alternatively be selectively
/// enabled in the regular expression itself via the `s` flag.
pub fn dot_matches_new_line(mut self, yes: bool) -> Config {
self.dot_matches_new_line = yes;
self
}
/// Enable or disable the "CRLF mode" flag by default.
///
/// By default this is disabled. It may alternatively be selectively
/// enabled in the regular expression itself via the `R` flag.
///
/// When CRLF mode is enabled, the following happens:
///
/// * Unless `dot_matches_new_line` is enabled, `.` will match any character
/// except for `\r` and `\n`.
/// * When `multi_line` mode is enabled, `^` and `$` will treat `\r\n`,
/// `\r` and `\n` as line terminators. And in particular, neither will
/// match between a `\r` and a `\n`.
pub fn crlf(mut self, yes: bool) -> Config {
self.crlf = yes;
self
}
/// Sets the line terminator for use with `(?u-s:.)` and `(?-us:.)`.
///
/// Namely, instead of `.` (by default) matching everything except for `\n`,
/// this will cause `.` to match everything except for the byte given.
///
/// If `.` is used in a context where Unicode mode is enabled and this byte
/// isn't ASCII, then an error will be returned. When Unicode mode is
/// disabled, then any byte is permitted, but will return an error if UTF-8
/// mode is enabled and it is a non-ASCII byte.
///
/// In short, any ASCII value for a line terminator is always okay. But a
/// non-ASCII byte might result in an error depending on whether Unicode
/// mode or UTF-8 mode are enabled.
///
/// Note that if `R` mode is enabled then it always takes precedence and
/// the line terminator will be treated as `\r` and `\n` simultaneously.
///
/// Note also that this *doesn't* impact the look-around assertions
/// `(?m:^)` and `(?m:$)`. That's usually controlled by additional
/// configuration in the regex engine itself.
pub fn line_terminator(mut self, byte: u8) -> Config {
self.line_terminator = byte;
self
}
/// Enable or disable the "swap greed" flag by default.
///
/// When this is enabled, `.*` (for example) will become ungreedy and `.*?`
/// will become greedy.
///
/// By default this is disabled. It may alternatively be selectively
/// enabled in the regular expression itself via the `U` flag.
pub fn swap_greed(mut self, yes: bool) -> Config {
self.swap_greed = yes;
self
}
/// Enable verbose mode in the regular expression.
///
/// When enabled, verbose mode permits insignificant whitespace in many
/// places in the regular expression, as well as comments. Comments are
/// started using `#` and continue until the end of the line.
///
/// By default, this is disabled. It may be selectively enabled in the
/// regular expression by using the `x` flag regardless of this setting.
pub fn ignore_whitespace(mut self, yes: bool) -> Config {
self.ignore_whitespace = yes;
self
}
/// Enable or disable the Unicode flag (`u`) by default.
///
/// By default this is **enabled**. It may alternatively be selectively
/// disabled in the regular expression itself via the `u` flag.
///
/// Note that unless "allow invalid UTF-8" is enabled (it's disabled by
/// default), a regular expression will fail to parse if Unicode mode is
/// disabled and a sub-expression could possibly match invalid UTF-8.
///
/// **WARNING**: Unicode mode can greatly increase the size of the compiled
/// DFA, which can noticeably impact both memory usage and compilation
/// time. This is especially noticeable if your regex contains character
/// classes like `\w` that are impacted by whether Unicode is enabled or
/// not. If Unicode is not necessary, you are encouraged to disable it.
pub fn unicode(mut self, yes: bool) -> Config {
self.unicode = yes;
self
}
/// When disabled, the builder will permit the construction of a regular
/// expression that may match invalid UTF-8.
///
/// For example, when [`Config::unicode`] is disabled, then
/// expressions like `[^a]` may match invalid UTF-8 since they can match
/// any single byte that is not `a`. By default, these sub-expressions
/// are disallowed to avoid returning offsets that split a UTF-8
/// encoded codepoint. However, in cases where matching at arbitrary
/// locations is desired, this option can be disabled to permit all such
/// sub-expressions.
///
/// When enabled (the default), the builder is guaranteed to produce a
/// regex that will only ever match valid UTF-8 (otherwise, the builder
/// will return an error).
pub fn utf8(mut self, yes: bool) -> Config {
self.utf8 = yes;
self
}
/// Set the nesting limit used for the regular expression parser.
///
/// The nesting limit controls how deep the abstract syntax tree is allowed
/// to be. If the AST exceeds the given limit (e.g., with too many nested
/// groups), then an error is returned by the parser.
///
/// The purpose of this limit is to act as a heuristic to prevent stack
/// overflow when building a finite automaton from a regular expression's
/// abstract syntax tree. In particular, construction currently uses
/// recursion. In the future, the implementation may stop using recursion
/// and this option will no longer be necessary.
///
/// This limit is not checked until the entire AST is parsed. Therefore,
/// if callers want to put a limit on the amount of heap space used, then
/// they should impose a limit on the length, in bytes, of the concrete
/// pattern string. In particular, this is viable since the parser will
/// limit itself to heap space proportional to the length of the pattern
/// string.
///
/// Note that a nest limit of `0` will return a nest limit error for most
/// patterns but not all. For example, a nest limit of `0` permits `a` but
/// not `ab`, since `ab` requires a concatenation AST item, which results
/// in a nest depth of `1`. In general, a nest limit is not something that
/// manifests in an obvious way in the concrete syntax, therefore, it
/// should not be used in a granular way.
pub fn nest_limit(mut self, limit: u32) -> Config {
self.nest_limit = limit;
self
}
/// Whether to support octal syntax or not.
///
/// Octal syntax is a little-known way of uttering Unicode codepoints in
/// a regular expression. For example, `a`, `\x61`, `\u0061` and
/// `\141` are all equivalent regular expressions, where the last example
/// shows octal syntax.
///
/// While supporting octal syntax isn't in and of itself a problem, it does
/// make good error messages harder. That is, in PCRE based regex engines,
/// syntax like `\1` invokes a backreference, which is explicitly
/// unsupported in Rust's regex engine. However, many users expect it to
/// be supported. Therefore, when octal support is disabled, the error
/// message will explicitly mention that backreferences aren't supported.
///
/// Octal syntax is disabled by default.
pub fn octal(mut self, yes: bool) -> Config {
self.octal = yes;
self
}
/// Returns whether "unicode" mode is enabled.
pub fn get_unicode(&self) -> bool {
self.unicode
}
/// Returns whether "case insensitive" mode is enabled.
pub fn get_case_insensitive(&self) -> bool {
self.case_insensitive
}
/// Returns whether "multi line" mode is enabled.
pub fn get_multi_line(&self) -> bool {
self.multi_line
}
/// Returns whether "dot matches new line" mode is enabled.
pub fn get_dot_matches_new_line(&self) -> bool {
self.dot_matches_new_line
}
/// Returns whether "CRLF" mode is enabled.
pub fn get_crlf(&self) -> bool {
self.crlf
}
/// Returns the line terminator in this syntax configuration.
pub fn get_line_terminator(&self) -> u8 {
self.line_terminator
}
/// Returns whether "swap greed" mode is enabled.
pub fn get_swap_greed(&self) -> bool {
self.swap_greed
}
/// Returns whether "ignore whitespace" mode is enabled.
pub fn get_ignore_whitespace(&self) -> bool {
self.ignore_whitespace
}
/// Returns whether UTF-8 mode is enabled.
pub fn get_utf8(&self) -> bool {
self.utf8
}
/// Returns the "nest limit" setting.
pub fn get_nest_limit(&self) -> u32 {
self.nest_limit
}
/// Returns whether "octal" mode is enabled.
pub fn get_octal(&self) -> bool {
self.octal
}
/// Applies this configuration to the given parser.
pub(crate) fn apply(&self, builder: &mut ParserBuilder) {
builder
.unicode(self.unicode)
.case_insensitive(self.case_insensitive)
.multi_line(self.multi_line)
.dot_matches_new_line(self.dot_matches_new_line)
.crlf(self.crlf)
.line_terminator(self.line_terminator)
.swap_greed(self.swap_greed)
.ignore_whitespace(self.ignore_whitespace)
.utf8(self.utf8)
.nest_limit(self.nest_limit)
.octal(self.octal);
}
/// Applies this configuration to the given AST parser.
pub(crate) fn apply_ast(&self, builder: &mut ast::parse::ParserBuilder) {
builder
.ignore_whitespace(self.ignore_whitespace)
.nest_limit(self.nest_limit)
.octal(self.octal);
}
/// Applies this configuration to the given AST-to-HIR translator.
pub(crate) fn apply_hir(
&self,
builder: &mut hir::translate::TranslatorBuilder,
) {
builder
.unicode(self.unicode)
.case_insensitive(self.case_insensitive)
.multi_line(self.multi_line)
.crlf(self.crlf)
.dot_matches_new_line(self.dot_matches_new_line)
.line_terminator(self.line_terminator)
.swap_greed(self.swap_greed)
.utf8(self.utf8);
}
}
impl Default for Config {
fn default() -> Config {
Config::new()
}
}
third_party/crates/regex-automata-0.4.18/src/util/unicode_data/mod.rs
Vendored
+17
View File
@@ -0,0 +1,17 @@
// This cfg should match the one in src/util/look.rs that uses perl_word.
#[cfg(all(
// We have to explicitly want to support Unicode word boundaries.
feature = "unicode-word-boundary",
not(all(
// If we don't have regex-syntax at all, then we definitely need to
// bring our own \w data table.
feature = "syntax",
// If unicode-perl is enabled, then regex-syntax/unicode-perl is
// also enabled, which in turn means we can use regex-syntax's
// is_word_character routine (and thus use its data tables). But if
// unicode-perl is not enabled, even if syntax is, then we need to
// bring our own.
feature = "unicode-perl",
)),
))]
pub(crate) mod perl_word;
third_party/crates/regex-automata-0.4.18/src/util/unicode_data/perl_word.rs
Vendored
+806
View File
@@ -0,0 +1,806 @@
// DO NOT EDIT THIS FILE. IT WAS AUTOMATICALLY GENERATED BY:
//
// ucd-generate perl-word ucd-16.0.0 --chars
//
// Unicode version: 16.0.0.
//
// ucd-generate 0.3.1 is available on crates.io.
pub const PERL_WORD: &'static [(char, char)] = &[
('0', '9'),
('A', 'Z'),
('_', '_'),
('a', 'z'),
('ª', 'ª'),
('µ', 'µ'),
('º', 'º'),
('À', 'Ö'),
('Ø', 'ö'),
('ø', 'ˁ'),
('ˆ', 'ˑ'),
('ˠ', 'ˤ'),
('ˬ', 'ˬ'),
('ˮ', 'ˮ'),
('\u{300}', 'ʹ'),
('Ͷ', 'ͷ'),
('ͺ', 'ͽ'),
('Ϳ', 'Ϳ'),
('Ά', 'Ά'),
('Έ', 'Ί'),
('Ό', 'Ό'),
('Ύ', 'Ρ'),
('Σ', 'ϵ'),
('Ϸ', 'ҁ'),
('\u{483}', 'ԯ'),
('Ա', 'Ֆ'),
('ՙ', 'ՙ'),
('ՠ', 'ֈ'),
('\u{591}', '\u{5bd}'),
('\u{5bf}', '\u{5bf}'),
('\u{5c1}', '\u{5c2}'),
('\u{5c4}', '\u{5c5}'),
('\u{5c7}', '\u{5c7}'),
('א', 'ת'),
('ׯ', 'ײ'),
('\u{610}', '\u{61a}'),
('ؠ', '٩'),
('ٮ', 'ۓ'),
('ە', '\u{6dc}'),
('\u{6df}', '\u{6e8}'),
('\u{6ea}', 'ۼ'),
('ۿ', 'ۿ'),
('ܐ', '\u{74a}'),
('ݍ', 'ޱ'),
('߀', 'ߵ'),
('ߺ', 'ߺ'),
('\u{7fd}', '\u{7fd}'),
('ࠀ', '\u{82d}'),
('ࡀ', '\u{85b}'),
('ࡠ', 'ࡪ'),
('ࡰ', 'ࢇ'),
('ࢉ', 'ࢎ'),
('\u{897}', '\u{8e1}'),
('\u{8e3}', '\u{963}'),
('०', '९'),
('ॱ', 'ঃ'),
('অ', 'ঌ'),
('এ', 'ঐ'),
('ও', 'ন'),
('প', 'র'),
('ল', 'ল'),
('শ', 'হ'),
('\u{9bc}', '\u{9c4}'),
('ে', 'ৈ'),
('ো', 'ৎ'),
('\u{9d7}', '\u{9d7}'),
('ড়', 'ঢ়'),
('য়', '\u{9e3}'),
('০', 'ৱ'),
('ৼ', 'ৼ'),
('\u{9fe}', '\u{9fe}'),
('\u{a01}', 'ਃ'),
('ਅ', 'ਊ'),
('ਏ', 'ਐ'),
('ਓ', 'ਨ'),
('ਪ', 'ਰ'),
('ਲ', 'ਲ਼'),
('ਵ', 'ਸ਼'),
('ਸ', 'ਹ'),
('\u{a3c}', '\u{a3c}'),
('ਾ', '\u{a42}'),
('\u{a47}', '\u{a48}'),
('\u{a4b}', '\u{a4d}'),
('\u{a51}', '\u{a51}'),
('ਖ਼', 'ੜ'),
('ਫ਼', 'ਫ਼'),
('੦', '\u{a75}'),
('\u{a81}', 'ઃ'),
('અ', 'ઍ'),
('એ', 'ઑ'),
('ઓ', 'ન'),
('પ', 'ર'),
('લ', 'ળ'),
('વ', 'હ'),
('\u{abc}', '\u{ac5}'),
('\u{ac7}', 'ૉ'),
('ો', '\u{acd}'),
('ૐ', 'ૐ'),
('ૠ', '\u{ae3}'),
('૦', '૯'),
('ૹ', '\u{aff}'),
('\u{b01}', 'ଃ'),
('ଅ', 'ଌ'),
('ଏ', 'ଐ'),
('ଓ', 'ନ'),
('ପ', 'ର'),
('ଲ', 'ଳ'),
('ଵ', 'ହ'),
('\u{b3c}', '\u{b44}'),
('େ', 'ୈ'),
('ୋ', '\u{b4d}'),
('\u{b55}', '\u{b57}'),
('ଡ଼', 'ଢ଼'),
('ୟ', '\u{b63}'),
('୦', '୯'),
('ୱ', 'ୱ'),
('\u{b82}', 'ஃ'),
('அ', 'ஊ'),
('எ', 'ஐ'),
('ஒ', 'க'),
('ங', 'ச'),
('ஜ', 'ஜ'),
('ஞ', 'ட'),
('ண', 'த'),
('ந', 'ப'),
('ம', 'ஹ'),
('\u{bbe}', 'ூ'),
('ெ', 'ை'),
('ொ', '\u{bcd}'),
('ௐ', 'ௐ'),
('\u{bd7}', '\u{bd7}'),
('௦', '௯'),
('\u{c00}', 'ఌ'),
('ఎ', 'ఐ'),
('ఒ', 'న'),
('ప', 'హ'),
('\u{c3c}', 'ౄ'),
('\u{c46}', '\u{c48}'),
('\u{c4a}', '\u{c4d}'),
('\u{c55}', '\u{c56}'),
('ౘ', 'ౚ'),
('ౝ', 'ౝ'),
('ౠ', '\u{c63}'),
('౦', '౯'),
('ಀ', 'ಃ'),
('ಅ', 'ಌ'),
('ಎ', 'ಐ'),
('ಒ', 'ನ'),
('ಪ', 'ಳ'),
('ವ', 'ಹ'),
('\u{cbc}', 'ೄ'),
('\u{cc6}', '\u{cc8}'),
('\u{cca}', '\u{ccd}'),
('\u{cd5}', '\u{cd6}'),
('ೝ', 'ೞ'),
('ೠ', '\u{ce3}'),
('೦', '೯'),
('ೱ', 'ೳ'),
('\u{d00}', 'ഌ'),
('എ', 'ഐ'),
('ഒ', '\u{d44}'),
('െ', 'ൈ'),
('ൊ', 'ൎ'),
('ൔ', '\u{d57}'),
('ൟ', '\u{d63}'),
('൦', '൯'),
('ൺ', 'ൿ'),
('\u{d81}', 'ඃ'),
('අ', 'ඖ'),
('ක', 'න'),
('ඳ', 'ර'),
('ල', 'ල'),
('ව', 'ෆ'),
('\u{dca}', '\u{dca}'),
('\u{dcf}', '\u{dd4}'),
('\u{dd6}', '\u{dd6}'),
('ෘ', '\u{ddf}'),
('෦', '෯'),
('ෲ', 'ෳ'),
('ก', '\u{e3a}'),
('เ', '\u{e4e}'),
('๐', '๙'),
('ກ', 'ຂ'),
('ຄ', 'ຄ'),
('ຆ', 'ຊ'),
('ຌ', 'ຣ'),
('ລ', 'ລ'),
('ວ', 'ຽ'),
('ເ', 'ໄ'),
('ໆ', 'ໆ'),
('\u{ec8}', '\u{ece}'),
('໐', '໙'),
('ໜ', 'ໟ'),
('ༀ', 'ༀ'),
('\u{f18}', '\u{f19}'),
('༠', '༩'),
('\u{f35}', '\u{f35}'),
('\u{f37}', '\u{f37}'),
('\u{f39}', '\u{f39}'),
('༾', 'ཇ'),
('ཉ', 'ཬ'),
('\u{f71}', '\u{f84}'),
('\u{f86}', '\u{f97}'),
('\u{f99}', '\u{fbc}'),
('\u{fc6}', '\u{fc6}'),
('က', '၉'),
('ၐ', '\u{109d}'),
('Ⴀ', 'Ⴥ'),
('Ⴧ', 'Ⴧ'),
('Ⴭ', 'Ⴭ'),
('ა', 'ჺ'),
('ჼ', 'ቈ'),
('ቊ', 'ቍ'),
('ቐ', 'ቖ'),
('ቘ', 'ቘ'),
('ቚ', 'ቝ'),
('በ', 'ኈ'),
('ኊ', 'ኍ'),
('ነ', 'ኰ'),
('ኲ', 'ኵ'),
('ኸ', 'ኾ'),
('ዀ', 'ዀ'),
('ዂ', 'ዅ'),
('ወ', 'ዖ'),
('ዘ', 'ጐ'),
('ጒ', 'ጕ'),
('ጘ', 'ፚ'),
('\u{135d}', '\u{135f}'),
('ᎀ', 'ᎏ'),
('Ꭰ', 'Ᏽ'),
('ᏸ', 'ᏽ'),
('ᐁ', 'ᙬ'),
('ᙯ', 'ᙿ'),
('ᚁ', 'ᚚ'),
('ᚠ', 'ᛪ'),
('ᛮ', 'ᛸ'),
('ᜀ', '\u{1715}'),
('ᜟ', '\u{1734}'),
('ᝀ', '\u{1753}'),
('ᝠ', 'ᝬ'),
('ᝮ', 'ᝰ'),
('\u{1772}', '\u{1773}'),
('ក', '\u{17d3}'),
('ៗ', 'ៗ'),
('ៜ', '\u{17dd}'),
('០', '៩'),
('\u{180b}', '\u{180d}'),
('\u{180f}', '᠙'),
('ᠠ', 'ᡸ'),
('ᢀ', 'ᢪ'),
('ᢰ', 'ᣵ'),
('ᤀ', 'ᤞ'),
('\u{1920}', 'ᤫ'),
('ᤰ', '\u{193b}'),
('᥆', 'ᥭ'),
('ᥰ', 'ᥴ'),
('ᦀ', 'ᦫ'),
('ᦰ', 'ᧉ'),
('᧐', '᧙'),
('ᨀ', '\u{1a1b}'),
('ᨠ', '\u{1a5e}'),
('\u{1a60}', '\u{1a7c}'),
('\u{1a7f}', '᪉'),
('᪐', '᪙'),
('ᪧ', 'ᪧ'),
('\u{1ab0}', '\u{1ace}'),
('\u{1b00}', 'ᭌ'),
('᭐', '᭙'),
('\u{1b6b}', '\u{1b73}'),
('\u{1b80}', '\u{1bf3}'),
('ᰀ', '\u{1c37}'),
('᱀', '᱉'),
('ᱍ', 'ᱽ'),
('ᲀ', 'ᲊ'),
('Ა', 'Ჺ'),
('Ჽ', 'Ჿ'),
('\u{1cd0}', '\u{1cd2}'),
('\u{1cd4}', 'ᳺ'),
('ᴀ', 'ἕ'),
('Ἐ', 'Ἕ'),
('ἠ', 'ὅ'),
('Ὀ', 'Ὅ'),
('ὐ', 'ὗ'),
('Ὑ', 'Ὑ'),
('Ὓ', 'Ὓ'),
('Ὕ', 'Ὕ'),
('Ὗ', 'ώ'),
('ᾀ', 'ᾴ'),
('ᾶ', 'ᾼ'),
('ι', 'ι'),
('ῂ', 'ῄ'),
('ῆ', 'ῌ'),
('ῐ', 'ΐ'),
('ῖ', 'Ί'),
('ῠ', 'Ῥ'),
('ῲ', 'ῴ'),
('ῶ', 'ῼ'),
('\u{200c}', '\u{200d}'),
('‿', '⁀'),
('⁔', '⁔'),
('ⁱ', 'ⁱ'),
('ⁿ', 'ⁿ'),
('ₐ', 'ₜ'),
('\u{20d0}', '\u{20f0}'),
('ℂ', 'ℂ'),
('ℇ', 'ℇ'),
('ℊ', 'ℓ'),
('ℕ', 'ℕ'),
('ℙ', 'ℝ'),
('ℤ', 'ℤ'),
('Ω', 'Ω'),
('ℨ', 'ℨ'),
('K', 'ℭ'),
('ℯ', 'ℹ'),
('ℼ', 'ℿ'),
('ⅅ', 'ⅉ'),
('ⅎ', 'ⅎ'),
('Ⅰ', 'ↈ'),
('Ⓐ', 'ⓩ'),
('Ⰰ', 'ⳤ'),
('Ⳬ', 'ⳳ'),
('ⴀ', 'ⴥ'),
('ⴧ', 'ⴧ'),
('ⴭ', 'ⴭ'),
('ⴰ', 'ⵧ'),
('ⵯ', 'ⵯ'),
('\u{2d7f}', 'ⶖ'),
('ⶠ', 'ⶦ'),
('ⶨ', 'ⶮ'),
('ⶰ', 'ⶶ'),
('ⶸ', 'ⶾ'),
('ⷀ', 'ⷆ'),
('ⷈ', 'ⷎ'),
('ⷐ', 'ⷖ'),
('ⷘ', 'ⷞ'),
('\u{2de0}', '\u{2dff}'),
('ⸯ', 'ⸯ'),
('々', '〇'),
('〡', '\u{302f}'),
('〱', '〵'),
('〸', '〼'),
('ぁ', 'ゖ'),
('\u{3099}', '\u{309a}'),
('ゝ', 'ゟ'),
('ァ', 'ヺ'),
('ー', 'ヿ'),
('ㄅ', 'ㄯ'),
('ㄱ', 'ㆎ'),
('ㆠ', 'ㆿ'),
('ㇰ', 'ㇿ'),
('㐀', '䶿'),
('一', 'ꒌ'),
('ꓐ', 'ꓽ'),
('ꔀ', 'ꘌ'),
('ꘐ', 'ꘫ'),
('Ꙁ', '\u{a672}'),
('\u{a674}', '\u{a67d}'),
('ꙿ', '\u{a6f1}'),
('ꜗ', 'ꜟ'),
('Ꜣ', 'ꞈ'),
('Ꞌ', 'ꟍ'),
('Ꟑ', 'ꟑ'),
('ꟓ', 'ꟓ'),
('ꟕ', 'Ƛ'),
('ꟲ', 'ꠧ'),
('\u{a82c}', '\u{a82c}'),
('ꡀ', 'ꡳ'),
('ꢀ', '\u{a8c5}'),
('꣐', '꣙'),
('\u{a8e0}', 'ꣷ'),
('ꣻ', 'ꣻ'),
('ꣽ', '\u{a92d}'),
('ꤰ', '\u{a953}'),
('ꥠ', 'ꥼ'),
('\u{a980}', '\u{a9c0}'),
('ꧏ', '꧙'),
('ꧠ', 'ꧾ'),
('ꨀ', '\u{aa36}'),
('ꩀ', 'ꩍ'),
('꩐', '꩙'),
('ꩠ', 'ꩶ'),
('ꩺ', 'ꫂ'),
('ꫛ', 'ꫝ'),
('ꫠ', 'ꫯ'),
('ꫲ', '\u{aaf6}'),
('ꬁ', 'ꬆ'),
('ꬉ', 'ꬎ'),
('ꬑ', 'ꬖ'),
('ꬠ', 'ꬦ'),
('ꬨ', 'ꬮ'),
('ꬰ', 'ꭚ'),
('ꭜ', 'ꭩ'),
('ꭰ', 'ꯪ'),
('꯬', '\u{abed}'),
('꯰', '꯹'),
('가', '힣'),
('ힰ', 'ퟆ'),
('ퟋ', 'ퟻ'),
('豈', '舘'),
('並', '龎'),
('ff', 'st'),
('ﬓ', 'ﬗ'),
('יִ', 'ﬨ'),
('שׁ', 'זּ'),
('טּ', 'לּ'),
('מּ', 'מּ'),
('נּ', 'סּ'),
('ףּ', 'פּ'),
('צּ', 'ﮱ'),
('ﯓ', 'ﴽ'),
('ﵐ', 'ﶏ'),
('ﶒ', 'ﷇ'),
('ﷰ', 'ﷻ'),
('\u{fe00}', '\u{fe0f}'),
('\u{fe20}', '\u{fe2f}'),
('︳', '︴'),
('﹍', '﹏'),
('ﹰ', 'ﹴ'),
('ﹶ', 'ﻼ'),
('０', '９'),
('Ａ', 'Ｚ'),
('＿', '＿'),
('ａ', 'ｚ'),
('ｦ', 'ﾾ'),
('ￂ', 'ￇ'),
('ￊ', 'ￏ'),
('ￒ', 'ￗ'),
('ￚ', 'ￜ'),
('𐀀', '𐀋'),
('𐀍', '𐀦'),
('𐀨', '𐀺'),
('𐀼', '𐀽'),
('𐀿', '𐁍'),
('𐁐', '𐁝'),
('𐂀', '𐃺'),
('𐅀', '𐅴'),
('\u{101fd}', '\u{101fd}'),
('𐊀', '𐊜'),
('𐊠', '𐋐'),
('\u{102e0}', '\u{102e0}'),
('𐌀', '𐌟'),
('𐌭', '𐍊'),
('𐍐', '\u{1037a}'),
('𐎀', '𐎝'),
('𐎠', '𐏃'),
('𐏈', '𐏏'),
('𐏑', '𐏕'),
('𐐀', '𐒝'),
('𐒠', '𐒩'),
('𐒰', '𐓓'),
('𐓘', '𐓻'),
('𐔀', '𐔧'),
('𐔰', '𐕣'),
('𐕰', '𐕺'),
('𐕼', '𐖊'),
('𐖌', '𐖒'),
('𐖔', '𐖕'),
('𐖗', '𐖡'),
('𐖣', '𐖱'),
('𐖳', '𐖹'),
('𐖻', '𐖼'),
('𐗀', '𐗳'),
('𐘀', '𐜶'),
('𐝀', '𐝕'),
('𐝠', '𐝧'),
('𐞀', '𐞅'),
('𐞇', '𐞰'),
('𐞲', '𐞺'),
('𐠀', '𐠅'),
('𐠈', '𐠈'),
('𐠊', '𐠵'),
('𐠷', '𐠸'),
('𐠼', '𐠼'),
('𐠿', '𐡕'),
('𐡠', '𐡶'),
('𐢀', '𐢞'),
('𐣠', '𐣲'),
('𐣴', '𐣵'),
('𐤀', '𐤕'),
('𐤠', '𐤹'),
('𐦀', '𐦷'),
('𐦾', '𐦿'),
('𐨀', '\u{10a03}'),
('\u{10a05}', '\u{10a06}'),
('\u{10a0c}', '𐨓'),
('𐨕', '𐨗'),
('𐨙', '𐨵'),
('\u{10a38}', '\u{10a3a}'),
('\u{10a3f}', '\u{10a3f}'),
('𐩠', '𐩼'),
('𐪀', '𐪜'),
('𐫀', '𐫇'),
('𐫉', '\u{10ae6}'),
('𐬀', '𐬵'),
('𐭀', '𐭕'),
('𐭠', '𐭲'),
('𐮀', '𐮑'),
('𐰀', '𐱈'),
('𐲀', '𐲲'),
('𐳀', '𐳲'),
('𐴀', '\u{10d27}'),
('𐴰', '𐴹'),
('𐵀', '𐵥'),
('\u{10d69}', '\u{10d6d}'),
('𐵯', '𐶅'),
('𐺀', '𐺩'),
('\u{10eab}', '\u{10eac}'),
('𐺰', '𐺱'),
('𐻂', '𐻄'),
('\u{10efc}', '𐼜'),
('𐼧', '𐼧'),
('𐼰', '\u{10f50}'),
('𐽰', '\u{10f85}'),
('𐾰', '𐿄'),
('𐿠', '𐿶'),
('𑀀', '\u{11046}'),
('𑁦', '𑁵'),
('\u{1107f}', '\u{110ba}'),
('\u{110c2}', '\u{110c2}'),
('𑃐', '𑃨'),
('𑃰', '𑃹'),
('\u{11100}', '\u{11134}'),
('𑄶', '𑄿'),
('𑅄', '𑅇'),
('𑅐', '\u{11173}'),
('𑅶', '𑅶'),
('\u{11180}', '𑇄'),
('\u{111c9}', '\u{111cc}'),
('𑇎', '𑇚'),
('𑇜', '𑇜'),
('𑈀', '𑈑'),
('𑈓', '\u{11237}'),
('\u{1123e}', '\u{11241}'),
('𑊀', '𑊆'),
('𑊈', '𑊈'),
('𑊊', '𑊍'),
('𑊏', '𑊝'),
('𑊟', '𑊨'),
('𑊰', '\u{112ea}'),
('𑋰', '𑋹'),
('\u{11300}', '𑌃'),
('𑌅', '𑌌'),
('𑌏', '𑌐'),
('𑌓', '𑌨'),
('𑌪', '𑌰'),
('𑌲', '𑌳'),
('𑌵', '𑌹'),
('\u{1133b}', '𑍄'),
('𑍇', '𑍈'),
('𑍋', '\u{1134d}'),
('𑍐', '𑍐'),
('\u{11357}', '\u{11357}'),
('𑍝', '𑍣'),
('\u{11366}', '\u{1136c}'),
('\u{11370}', '\u{11374}'),
('𑎀', '𑎉'),
('𑎋', '𑎋'),
('𑎎', '𑎎'),
('𑎐', '𑎵'),
('𑎷', '\u{113c0}'),
('\u{113c2}', '\u{113c2}'),
('\u{113c5}', '\u{113c5}'),
('\u{113c7}', '𑏊'),
('𑏌', '𑏓'),
('\u{113e1}', '\u{113e2}'),
('𑐀', '𑑊'),
('𑑐', '𑑙'),
('\u{1145e}', '𑑡'),
('𑒀', '𑓅'),
('𑓇', '𑓇'),
('𑓐', '𑓙'),
('𑖀', '\u{115b5}'),
('𑖸', '\u{115c0}'),
('𑗘', '\u{115dd}'),
('𑘀', '\u{11640}'),
('𑙄', '𑙄'),
('𑙐', '𑙙'),
('𑚀', '𑚸'),
('𑛀', '𑛉'),
('𑛐', '𑛣'),
('𑜀', '𑜚'),
('\u{1171d}', '\u{1172b}'),
('𑜰', '𑜹'),
('𑝀', '𑝆'),
('𑠀', '\u{1183a}'),
('𑢠', '𑣩'),
('𑣿', '𑤆'),
('𑤉', '𑤉'),
('𑤌', '𑤓'),
('𑤕', '𑤖'),
('𑤘', '𑤵'),
('𑤷', '𑤸'),
('\u{1193b}', '\u{11943}'),
('𑥐', '𑥙'),
('𑦠', '𑦧'),
('𑦪', '\u{119d7}'),
('\u{119da}', '𑧡'),
('𑧣', '𑧤'),
('𑨀', '\u{11a3e}'),
('\u{11a47}', '\u{11a47}'),
('𑩐', '\u{11a99}'),
('𑪝', '𑪝'),
('𑪰', '𑫸'),
('𑯀', '𑯠'),
('𑯰', '𑯹'),
('𑰀', '𑰈'),
('𑰊', '\u{11c36}'),
('\u{11c38}', '𑱀'),
('𑱐', '𑱙'),
('𑱲', '𑲏'),
('\u{11c92}', '\u{11ca7}'),
('𑲩', '\u{11cb6}'),
('𑴀', '𑴆'),
('𑴈', '𑴉'),
('𑴋', '\u{11d36}'),
('\u{11d3a}', '\u{11d3a}'),
('\u{11d3c}', '\u{11d3d}'),
('\u{11d3f}', '\u{11d47}'),
('𑵐', '𑵙'),
('𑵠', '𑵥'),
('𑵧', '𑵨'),
('𑵪', '𑶎'),
('\u{11d90}', '\u{11d91}'),
('𑶓', '𑶘'),
('𑶠', '𑶩'),
('𑻠', '𑻶'),
('\u{11f00}', '𑼐'),
('𑼒', '\u{11f3a}'),
('𑼾', '\u{11f42}'),
('𑽐', '\u{11f5a}'),
('𑾰', '𑾰'),
('𒀀', '𒎙'),
('𒐀', '𒑮'),
('𒒀', '𒕃'),
('𒾐', '𒿰'),
('𓀀', '𓐯'),
('\u{13440}', '\u{13455}'),
('𓑠', '𔏺'),
('𔐀', '𔙆'),
('𖄀', '𖄹'),
('𖠀', '𖨸'),
('𖩀', '𖩞'),
('𖩠', '𖩩'),
('𖩰', '𖪾'),
('𖫀', '𖫉'),
('𖫐', '𖫭'),
('\u{16af0}', '\u{16af4}'),
('𖬀', '\u{16b36}'),
('𖭀', '𖭃'),
('𖭐', '𖭙'),
('𖭣', '𖭷'),
('𖭽', '𖮏'),
('𖵀', '𖵬'),
('𖵰', '𖵹'),
('𖹀', '𖹿'),
('𖼀', '𖽊'),
('\u{16f4f}', '𖾇'),
('\u{16f8f}', '𖾟'),
('𖿠', '𖿡'),
('𖿣', '\u{16fe4}'),
('\u{16ff0}', '\u{16ff1}'),
('𗀀', '𘟷'),
('𘠀', '𘳕'),
('𘳿', '𘴈'),
('𚿰', '𚿳'),
('𚿵', '𚿻'),
('𚿽', '𚿾'),
('𛀀', '𛄢'),
('𛄲', '𛄲'),
('𛅐', '𛅒'),
('𛅕', '𛅕'),
('𛅤', '𛅧'),
('𛅰', '𛋻'),
('𛰀', '𛱪'),
('𛱰', '𛱼'),
('𛲀', '𛲈'),
('𛲐', '𛲙'),
('\u{1bc9d}', '\u{1bc9e}'),
('𜳰', '𜳹'),
('\u{1cf00}', '\u{1cf2d}'),
('\u{1cf30}', '\u{1cf46}'),
('\u{1d165}', '\u{1d169}'),
('\u{1d16d}', '\u{1d172}'),
('\u{1d17b}', '\u{1d182}'),
('\u{1d185}', '\u{1d18b}'),
('\u{1d1aa}', '\u{1d1ad}'),
('\u{1d242}', '\u{1d244}'),
('𝐀', '𝑔'),
('𝑖', '𝒜'),
('𝒞', '𝒟'),
('𝒢', '𝒢'),
('𝒥', '𝒦'),
('𝒩', '𝒬'),
('𝒮', '𝒹'),
('𝒻', '𝒻'),
('𝒽', '𝓃'),
('𝓅', '𝔅'),
('𝔇', '𝔊'),
('𝔍', '𝔔'),
('𝔖', '𝔜'),
('𝔞', '𝔹'),
('𝔻', '𝔾'),
('𝕀', '𝕄'),
('𝕆', '𝕆'),
('𝕊', '𝕐'),
('𝕒', '𝚥'),
('𝚨', '𝛀'),
('𝛂', '𝛚'),
('𝛜', '𝛺'),
('𝛼', '𝜔'),
('𝜖', '𝜴'),
('𝜶', '𝝎'),
('𝝐', '𝝮'),
('𝝰', '𝞈'),
('𝞊', '𝞨'),
('𝞪', '𝟂'),
('𝟄', '𝟋'),
('𝟎', '𝟿'),
('\u{1da00}', '\u{1da36}'),
('\u{1da3b}', '\u{1da6c}'),
('\u{1da75}', '\u{1da75}'),
('\u{1da84}', '\u{1da84}'),
('\u{1da9b}', '\u{1da9f}'),
('\u{1daa1}', '\u{1daaf}'),
('𝼀', '𝼞'),
('𝼥', '𝼪'),
('\u{1e000}', '\u{1e006}'),
('\u{1e008}', '\u{1e018}'),
('\u{1e01b}', '\u{1e021}'),
('\u{1e023}', '\u{1e024}'),
('\u{1e026}', '\u{1e02a}'),
('𞀰', '𞁭'),
('\u{1e08f}', '\u{1e08f}'),
('𞄀', '𞄬'),
('\u{1e130}', '𞄽'),
('𞅀', '𞅉'),
('𞅎', '𞅎'),
('𞊐', '\u{1e2ae}'),
('𞋀', '𞋹'),
('𞓐', '𞓹'),
('𞗐', '𞗺'),
('𞟠', '𞟦'),
('𞟨', '𞟫'),
('𞟭', '𞟮'),
('𞟰', '𞟾'),
('𞠀', '𞣄'),
('\u{1e8d0}', '\u{1e8d6}'),
('𞤀', '𞥋'),
('𞥐', '𞥙'),
('𞸀', '𞸃'),
('𞸅', '𞸟'),
('𞸡', '𞸢'),
('𞸤', '𞸤'),
('𞸧', '𞸧'),
('𞸩', '𞸲'),
('𞸴', '𞸷'),
('𞸹', '𞸹'),
('𞸻', '𞸻'),
('𞹂', '𞹂'),
('𞹇', '𞹇'),
('𞹉', '𞹉'),
('𞹋', '𞹋'),
('𞹍', '𞹏'),
('𞹑', '𞹒'),
('𞹔', '𞹔'),
('𞹗', '𞹗'),
('𞹙', '𞹙'),
('𞹛', '𞹛'),
('𞹝', '𞹝'),
('𞹟', '𞹟'),
('𞹡', '𞹢'),
('𞹤', '𞹤'),
('𞹧', '𞹪'),
('𞹬', '𞹲'),
('𞹴', '𞹷'),
('𞹹', '𞹼'),
('𞹾', '𞹾'),
('𞺀', '𞺉'),
('𞺋', '𞺛'),
('𞺡', '𞺣'),
('𞺥', '𞺩'),
('𞺫', '𞺻'),
('🄰', '🅉'),
('🅐', '🅩'),
('🅰', '🆉'),
('🯰', '🯹'),
('𠀀', '𪛟'),
('𪜀', '𫜹'),
('𫝀', '𫠝'),
('𫠠', '𬺡'),
('𬺰', '𮯠'),
('𮯰', '𮹝'),
('丽', '𪘀'),
('𰀀', '𱍊'),
('𱍐', '𲎯'),
('\u{e0100}', '\u{e01ef}'),
];
third_party/crates/regex-automata-0.4.18/src/util/utf8.rs
Vendored
+191
View File
@@ -0,0 +1,191 @@
/*!
Utilities for dealing with UTF-8.
This module provides some UTF-8 related helper routines, including an
incremental decoder.
*/
/// Returns true if and only if the given byte is considered a word character.
/// This only applies to ASCII.
///
/// This was copied from regex-syntax so that we can use it to determine the
/// starting DFA state while searching without depending on regex-syntax. The
/// definition is never going to change, so there's no maintenance/bit-rot
/// hazard here.
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn is_word_byte(b: u8) -> bool {
const fn mkwordset() -> [bool; 256] {
// FIXME: Use as_usize() once const functions in traits are stable.
let mut set = [false; 256];
set[b'_' as usize] = true;
let mut byte = b'0';
while byte <= b'9' {
set[byte as usize] = true;
byte += 1;
}
byte = b'A';
while byte <= b'Z' {
set[byte as usize] = true;
byte += 1;
}
byte = b'a';
while byte <= b'z' {
set[byte as usize] = true;
byte += 1;
}
set
}
const WORD: [bool; 256] = mkwordset();
WORD[b as usize]
}
/// Decodes the next UTF-8 encoded codepoint from the given byte slice.
///
/// If no valid encoding of a codepoint exists at the beginning of the given
/// byte slice, then the first byte is returned instead.
///
/// This returns `None` if and only if `bytes` is empty.
///
/// This never panics.
///
/// *WARNING*: This is not designed for performance. If you're looking for a
/// fast UTF-8 decoder, this is not it. If you feel like you need one in this
/// crate, then please file an issue and discuss your use case.
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn decode(bytes: &[u8]) -> Option<Result<char, u8>> {
if bytes.is_empty() {
return None;
}
let len = match len(bytes[0]) {
None => return Some(Err(bytes[0])),
Some(len) if len > bytes.len() => return Some(Err(bytes[0])),
Some(1) => return Some(Ok(char::from(bytes[0]))),
Some(len) => len,
};
match core::str::from_utf8(&bytes[..len]) {
Ok(s) => Some(Ok(s.chars().next().unwrap())),
Err(_) => Some(Err(bytes[0])),
}
}
/// Decodes the last UTF-8 encoded codepoint from the given byte slice.
///
/// If no valid encoding of a codepoint exists at the end of the given byte
/// slice, then the last byte is returned instead.
///
/// This returns `None` if and only if `bytes` is empty.
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn decode_last(bytes: &[u8]) -> Option<Result<char, u8>> {
if bytes.is_empty() {
return None;
}
let mut start = bytes.len() - 1;
let limit = bytes.len().saturating_sub(4);
while start > limit && !is_leading_or_invalid_byte(bytes[start]) {
start -= 1;
}
match decode(&bytes[start..]) {
None => None,
Some(Ok(ch)) => Some(Ok(ch)),
Some(Err(_)) => Some(Err(bytes[bytes.len() - 1])),
}
}
/// Given a UTF-8 leading byte, this returns the total number of code units
/// in the following encoded codepoint.
///
/// If the given byte is not a valid UTF-8 leading byte, then this returns
/// `None`.
#[cfg_attr(feature = "perf-inline", inline(always))]
fn len(byte: u8) -> Option<usize> {
match byte {
0b0000_0000..=0b0111_1111 => Some(1),
0b1000_0000..=0b1011_1111 => None,
0b1100_0000..=0b1101_1111 => Some(2),
0b1110_0000..=0b1110_1111 => Some(3),
0b1111_0000..=0b1111_0111 => Some(4),
_ => None,
}
}
/// Returns true if and only if the given offset in the given bytes falls on a
/// valid UTF-8 encoded codepoint boundary.
///
/// If `bytes` is not valid UTF-8, then the behavior of this routine is
/// unspecified.
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn is_boundary(bytes: &[u8], i: usize) -> bool {
match bytes.get(i) {
// The position at the end of the bytes always represents an empty
// string, which is a valid boundary. But anything after that doesn't
// make much sense to call valid a boundary.
None => i == bytes.len(),
// Other than ASCII (where the most significant bit is never set),
// valid starting bytes always have their most significant two bits
// set, where as continuation bytes never have their second most
// significant bit set. Therefore, this only returns true when bytes[i]
// corresponds to a byte that begins a valid UTF-8 encoding of a
// Unicode scalar value.
Some(&b) => b <= 0b0111_1111 || b >= 0b1100_0000,
}
}
/// Returns true if and only if the given byte is either a valid leading UTF-8
/// byte, or is otherwise an invalid byte that can never appear anywhere in a
/// valid UTF-8 sequence.
#[cfg_attr(feature = "perf-inline", inline(always))]
fn is_leading_or_invalid_byte(b: u8) -> bool {
// In the ASCII case, the most significant bit is never set. The leading
// byte of a 2/3/4-byte sequence always has the top two most significant
// bits set. For bytes that can never appear anywhere in valid UTF-8, this
// also returns true, since every such byte has its two most significant
// bits set:
//
// \xC0 :: 11000000
// \xC1 :: 11000001
// \xF5 :: 11110101
// \xF6 :: 11110110
// \xF7 :: 11110111
// \xF8 :: 11111000
// \xF9 :: 11111001
// \xFA :: 11111010
// \xFB :: 11111011
// \xFC :: 11111100
// \xFD :: 11111101
// \xFE :: 11111110
// \xFF :: 11111111
(b & 0b1100_0000) != 0b1000_0000
}
/*
/// Returns the smallest possible index of the next valid UTF-8 sequence
/// starting after `i`.
///
/// For all inputs, including invalid UTF-8 and any value of `i`, the return
/// value is guaranteed to be greater than `i`. (If there is no value greater
/// than `i` that fits in `usize`, then this panics.)
///
/// Generally speaking, this should only be called on `text` when it is
/// permitted to assume that it is valid UTF-8 and where either `i >=
/// text.len()` or where `text[i]` is a leading byte of a UTF-8 sequence.
///
/// NOTE: This method was used in a previous conception of iterators where we
/// specifically tried to skip over empty matches that split a codepoint by
/// simply requiring that our next search begin at the beginning of codepoint.
/// But we ended up changing that technique to always advance by 1 byte and
/// then filter out matches that split a codepoint after-the-fact. Thus, we no
/// longer use this method. But I've kept it around in case we want to switch
/// back to this approach. Its guarantees are a little subtle, so I'd prefer
/// not to rebuild it from whole cloth.
pub(crate) fn next(text: &[u8], i: usize) -> usize {
let b = match text.get(i) {
None => return i.checked_add(1).unwrap(),
Some(&b) => b,
};
// For cases where we see an invalid UTF-8 byte, there isn't much we can do
// other than just start at the next byte.
let inc = len(b).unwrap_or(1);
i.checked_add(inc).unwrap()
}
*/
third_party/crates/regex-automata-0.4.18/src/util/wire.rs
Vendored
+947
View File
@@ -0,0 +1,947 @@
/*!
Types and routines that support the wire format of finite automata.
Currently, this module just exports a few error types and some small helpers
for deserializing [dense DFAs](crate::dfa::dense::DFA) using correct alignment.
*/
/*
A collection of helper functions, types and traits for serializing automata.
This crate defines its own bespoke serialization mechanism for some structures
provided in the public API, namely, DFAs. A bespoke mechanism was developed
primarily because structures like automata demand a specific binary format.
Attempting to encode their rich structure in an existing serialization
format is just not feasible. Moreover, the format for each structure is
generally designed such that deserialization is cheap. More specifically, that
deserialization can be done in constant time. (The idea being that you can
embed it into your binary or mmap it, and then use it immediately.)
In order to achieve this, the dense and sparse DFAs in this crate use an
in-memory representation that very closely corresponds to its binary serialized
form. This pervades and complicates everything, and in some cases, requires
dealing with alignment and reasoning about safety.
This technique does have major advantages. In particular, it permits doing
the potentially costly work of compiling a finite state machine in an offline
manner, and then loading it at runtime not only without having to re-compile
the regex, but even without the code required to do the compilation. This, for
example, permits one to use a pre-compiled DFA not only in environments without
Rust's standard library, but also in environments without a heap.
In the code below, whenever we insert some kind of padding, it's to enforce a
4-byte alignment, unless otherwise noted. Namely, u32 is the only state ID type
supported. (In a previous version of this library, DFAs were generic over the
state ID representation.)
Also, serialization generally requires the caller to specify endianness,
where as deserialization always assumes native endianness (otherwise cheap
deserialization would be impossible). This implies that serializing a structure
generally requires serializing both its big-endian and little-endian variants,
and then loading the correct one based on the target's endianness.
*/
use core::{cmp, mem::size_of};
#[cfg(feature = "alloc")]
use alloc::{vec, vec::Vec};
use crate::util::{
int::Pointer,
primitives::{PatternID, PatternIDError, StateID, StateIDError},
};
/// A hack to align a smaller type `B` with a bigger type `T`.
///
/// The usual use of this is with `B = [u8]` and `T = u32`. That is,
/// it permits aligning a sequence of bytes on a 4-byte boundary. This
/// is useful in contexts where one wants to embed a serialized [dense
/// DFA](crate::dfa::dense::DFA) into a Rust a program while guaranteeing the
/// alignment required for the DFA.
///
/// See [`dense::DFA::from_bytes`](crate::dfa::dense::DFA::from_bytes) for an
/// example of how to use this type.
#[repr(C)]
#[derive(Debug)]
pub struct AlignAs<B: ?Sized, T> {
/// A zero-sized field indicating the alignment we want.
pub _align: [T; 0],
/// A possibly non-sized field containing a sequence of bytes.
pub bytes: B,
}
/// An error that occurs when serializing an object from this crate.
///
/// Serialization, as used in this crate, universally refers to the process
/// of transforming a structure (like a DFA) into a custom binary format
/// represented by `&[u8]`. To this end, serialization is generally infallible.
/// However, it can fail when caller provided buffer sizes are too small. When
/// that occurs, a serialization error is reported.
///
/// A `SerializeError` provides no introspection capabilities. Its only
/// supported operation is conversion to a human readable error message.
///
/// This error type implements the `std::error::Error` trait only when the
/// `std` feature is enabled. Otherwise, this type is defined in all
/// configurations.
#[derive(Debug)]
pub struct SerializeError {
/// The name of the thing that a buffer is too small for.
///
/// Currently, the only kind of serialization error is one that is
/// committed by a caller: providing a destination buffer that is too
/// small to fit the serialized object. This makes sense conceptually,
/// since every valid inhabitant of a type should be serializable.
///
/// This is somewhat exposed in the public API of this crate. For example,
/// the `to_bytes_{big,little}_endian` APIs return a `Vec<u8>` and are
/// guaranteed to never panic or error. This is only possible because the
/// implementation guarantees that it will allocate a `Vec<u8>` that is
/// big enough.
///
/// In summary, if a new serialization error kind needs to be added, then
/// it will need careful consideration.
what: &'static str,
}
impl SerializeError {
pub(crate) fn buffer_too_small(what: &'static str) -> SerializeError {
SerializeError { what }
}
}
impl core::fmt::Display for SerializeError {
fn fmt(&self, f: &mut core::fmt::Formatter) -> core::fmt::Result {
write!(f, "destination buffer is too small to write {}", self.what)
}
}
#[cfg(feature = "std")]
impl std::error::Error for SerializeError {}
/// An error that occurs when deserializing an object defined in this crate.
///
/// Serialization, as used in this crate, universally refers to the process
/// of transforming a structure (like a DFA) into a custom binary format
/// represented by `&[u8]`. Deserialization, then, refers to the process of
/// cheaply converting this binary format back to the object's in-memory
/// representation as defined in this crate. To the extent possible,
/// deserialization will report this error whenever this process fails.
///
/// A `DeserializeError` provides no introspection capabilities. Its only
/// supported operation is conversion to a human readable error message.
///
/// This error type implements the `std::error::Error` trait only when the
/// `std` feature is enabled. Otherwise, this type is defined in all
/// configurations.
#[derive(Debug)]
pub struct DeserializeError(DeserializeErrorKind);
#[derive(Debug)]
enum DeserializeErrorKind {
Generic { msg: &'static str },
BufferTooSmall { what: &'static str },
InvalidUsize { what: &'static str },
VersionMismatch { expected: u32, found: u32 },
EndianMismatch { expected: u32, found: u32 },
AlignmentMismatch { alignment: usize, address: usize },
LabelMismatch { expected: &'static str },
ArithmeticOverflow { what: &'static str },
PatternID { err: PatternIDError, what: &'static str },
StateID { err: StateIDError, what: &'static str },
}
impl DeserializeError {
pub(crate) fn generic(msg: &'static str) -> DeserializeError {
DeserializeError(DeserializeErrorKind::Generic { msg })
}
pub(crate) fn buffer_too_small(what: &'static str) -> DeserializeError {
DeserializeError(DeserializeErrorKind::BufferTooSmall { what })
}
fn invalid_usize(what: &'static str) -> DeserializeError {
DeserializeError(DeserializeErrorKind::InvalidUsize { what })
}
fn version_mismatch(expected: u32, found: u32) -> DeserializeError {
DeserializeError(DeserializeErrorKind::VersionMismatch {
expected,
found,
})
}
fn endian_mismatch(expected: u32, found: u32) -> DeserializeError {
DeserializeError(DeserializeErrorKind::EndianMismatch {
expected,
found,
})
}
fn alignment_mismatch(
alignment: usize,
address: usize,
) -> DeserializeError {
DeserializeError(DeserializeErrorKind::AlignmentMismatch {
alignment,
address,
})
}
fn label_mismatch(expected: &'static str) -> DeserializeError {
DeserializeError(DeserializeErrorKind::LabelMismatch { expected })
}
fn arithmetic_overflow(what: &'static str) -> DeserializeError {
DeserializeError(DeserializeErrorKind::ArithmeticOverflow { what })
}
fn pattern_id_error(
err: PatternIDError,
what: &'static str,
) -> DeserializeError {
DeserializeError(DeserializeErrorKind::PatternID { err, what })
}
pub(crate) fn state_id_error(
err: StateIDError,
what: &'static str,
) -> DeserializeError {
DeserializeError(DeserializeErrorKind::StateID { err, what })
}
}
#[cfg(feature = "std")]
impl std::error::Error for DeserializeError {}
impl core::fmt::Display for DeserializeError {
fn fmt(&self, f: &mut core::fmt::Formatter) -> core::fmt::Result {
use self::DeserializeErrorKind::*;
match self.0 {
Generic { msg } => write!(f, "{msg}"),
BufferTooSmall { what } => {
write!(f, "buffer is too small to read {what}")
}
InvalidUsize { what } => {
write!(f, "{what} is too big to fit in a usize")
}
VersionMismatch { expected, found } => write!(
f,
"unsupported version: \
expected version {expected} but found version {found}",
),
EndianMismatch { expected, found } => write!(
f,
"endianness mismatch: expected 0x{expected:X} but \
got 0x{found:X}. (Are you trying to load an object \
serialized with a different endianness?)",
),
AlignmentMismatch { alignment, address } => write!(
f,
"alignment mismatch: slice starts at address 0x{address:X}, \
which is not aligned to a {alignment} byte boundary",
),
LabelMismatch { expected } => write!(
f,
"label mismatch: start of serialized object should \
contain a NUL terminated {expected:?} label, but a different \
label was found",
),
ArithmeticOverflow { what } => {
write!(f, "arithmetic overflow for {what}")
}
PatternID { ref err, what } => {
write!(f, "failed to read pattern ID for {what}: {err}")
}
StateID { ref err, what } => {
write!(f, "failed to read state ID for {what}: {err}")
}
}
}
}
/// Safely converts a `&[u32]` to `&[StateID]` with zero cost.
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn u32s_to_state_ids(slice: &[u32]) -> &[StateID] {
// SAFETY: This is safe because StateID is defined to have the same memory
// representation as a u32 (it is repr(transparent)). While not every u32
// is a "valid" StateID, callers are not permitted to rely on the validity
// of StateIDs for memory safety. It can only lead to logical errors. (This
// is why StateID::new_unchecked is safe.)
unsafe {
core::slice::from_raw_parts(
slice.as_ptr().cast::<StateID>(),
slice.len(),
)
}
}
/// Safely converts a `&mut [u32]` to `&mut [StateID]` with zero cost.
pub(crate) fn u32s_to_state_ids_mut(slice: &mut [u32]) -> &mut [StateID] {
// SAFETY: This is safe because StateID is defined to have the same memory
// representation as a u32 (it is repr(transparent)). While not every u32
// is a "valid" StateID, callers are not permitted to rely on the validity
// of StateIDs for memory safety. It can only lead to logical errors. (This
// is why StateID::new_unchecked is safe.)
unsafe {
core::slice::from_raw_parts_mut(
slice.as_mut_ptr().cast::<StateID>(),
slice.len(),
)
}
}
/// Safely converts a `&[u32]` to `&[PatternID]` with zero cost.
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn u32s_to_pattern_ids(slice: &[u32]) -> &[PatternID] {
// SAFETY: This is safe because PatternID is defined to have the same
// memory representation as a u32 (it is repr(transparent)). While not
// every u32 is a "valid" PatternID, callers are not permitted to rely
// on the validity of PatternIDs for memory safety. It can only lead to
// logical errors. (This is why PatternID::new_unchecked is safe.)
unsafe {
core::slice::from_raw_parts(
slice.as_ptr().cast::<PatternID>(),
slice.len(),
)
}
}
/// Checks that the given slice has an alignment that matches `T`.
///
/// This is useful for checking that a slice has an appropriate alignment
/// before casting it to a &[T]. Note though that alignment is not itself
/// sufficient to perform the cast for any `T`.
pub(crate) fn check_alignment<T>(
slice: &[u8],
) -> Result<(), DeserializeError> {
let alignment = core::mem::align_of::<T>();
let address = slice.as_ptr().as_usize();
if address % alignment == 0 {
return Ok(());
}
Err(DeserializeError::alignment_mismatch(alignment, address))
}
/// Reads a possibly empty amount of padding, up to 7 bytes, from the beginning
/// of the given slice. All padding bytes must be NUL bytes.
///
/// This is useful because it can be theoretically necessary to pad the
/// beginning of a serialized object with NUL bytes to ensure that it starts
/// at a correctly aligned address. These padding bytes should come immediately
/// before the label.
///
/// This returns the number of bytes read from the given slice.
pub(crate) fn skip_initial_padding(slice: &[u8]) -> usize {
let mut nread = 0;
while nread < 7 && nread < slice.len() && slice[nread] == 0 {
nread += 1;
}
nread
}
/// Allocate a byte buffer of the given size, along with some initial padding
/// such that `buf[padding..]` has the same alignment as `T`, where the
/// alignment of `T` must be at most `8`. In particular, callers should treat
/// the first N bytes (second return value) as padding bytes that must not be
/// overwritten. In all cases, the following identity holds:
///
/// ```ignore
/// let (buf, padding) = alloc_aligned_buffer::<StateID>(SIZE);
/// assert_eq!(SIZE, buf[padding..].len());
/// ```
///
/// In practice, padding is often zero.
///
/// The requirement for `8` as a maximum here is somewhat arbitrary. In
/// practice, we never need anything bigger in this crate, and so this function
/// does some sanity asserts under the assumption of a max alignment of `8`.
#[cfg(feature = "alloc")]
pub(crate) fn alloc_aligned_buffer<T>(size: usize) -> (Vec<u8>, usize) {
// NOTE: This is a kludge because there's no easy way to allocate a Vec<u8>
// with an alignment guaranteed to be greater than 1. We could create a
// Vec<u32>, but this cannot be safely transmuted to a Vec<u8> without
// concern, since reallocing or dropping the Vec<u8> is UB (different
// alignment than the initial allocation). We could define a wrapper type
// to manage this for us, but it seems like more machinery than it's worth.
let buf = vec![0; size];
let align = core::mem::align_of::<T>();
let address = buf.as_ptr().as_usize();
if address % align == 0 {
return (buf, 0);
}
// Let's try this again. We have to create a totally new alloc with
// the maximum amount of bytes we might need. We can't just extend our
// pre-existing 'buf' because that might create a new alloc with a
// different alignment.
let extra = align - 1;
let mut buf = vec![0; size + extra];
let address = buf.as_ptr().as_usize();
// The code below handles the case where 'address' is aligned to T, so if
// we got lucky and 'address' is now aligned to T (when it previously
// wasn't), then we're done.
if address % align == 0 {
buf.truncate(size);
return (buf, 0);
}
let padding = ((address & !(align - 1)).checked_add(align).unwrap())
.checked_sub(address)
.unwrap();
assert!(padding <= 7, "padding of {padding} is bigger than 7");
assert!(
padding <= extra,
"padding of {padding} is bigger than extra {extra} bytes",
);
buf.truncate(size + padding);
assert_eq!(size + padding, buf.len());
assert_eq!(
0,
buf[padding..].as_ptr().as_usize() % align,
"expected end of initial padding to be aligned to {align}",
);
(buf, padding)
}
/// Reads a NUL terminated label starting at the beginning of the given slice.
///
/// If a NUL terminated label could not be found, then an error is returned.
/// Similarly, if a label is found but doesn't match the expected label, then
/// an error is returned.
///
/// Upon success, the total number of bytes read (including padding bytes) is
/// returned.
pub(crate) fn read_label(
slice: &[u8],
expected_label: &'static str,
) -> Result<usize, DeserializeError> {
// Set an upper bound on how many bytes we scan for a NUL. Since no label
// in this crate is longer than 256 bytes, if we can't find one within that
// range, then we have corrupted data.
let first_nul =
slice[..cmp::min(slice.len(), 256)].iter().position(|&b| b == 0);
let first_nul = match first_nul {
Some(first_nul) => first_nul,
None => {
return Err(DeserializeError::generic(
"could not find NUL terminated label \
at start of serialized object",
));
}
};
let len = first_nul + padding_len(first_nul);
if slice.len() < len {
return Err(DeserializeError::generic(
"could not find properly sized label at start of serialized object"
));
}
if expected_label.as_bytes() != &slice[..first_nul] {
return Err(DeserializeError::label_mismatch(expected_label));
}
Ok(len)
}
/// Writes the given label to the buffer as a NUL terminated string. The label
/// given must not contain NUL, otherwise this will panic. Similarly, the label
/// must not be longer than 255 bytes, otherwise this will panic.
///
/// Additional NUL bytes are written as necessary to ensure that the number of
/// bytes written is always a multiple of 4.
///
/// Upon success, the total number of bytes written (including padding) is
/// returned.
pub(crate) fn write_label(
label: &str,
dst: &mut [u8],
) -> Result<usize, SerializeError> {
let nwrite = write_label_len(label);
if dst.len() < nwrite {
return Err(SerializeError::buffer_too_small("label"));
}
dst[..label.len()].copy_from_slice(label.as_bytes());
for i in 0..(nwrite - label.len()) {
dst[label.len() + i] = 0;
}
assert_eq!(nwrite % 4, 0);
Ok(nwrite)
}
/// Returns the total number of bytes (including padding) that would be written
/// for the given label. This panics if the given label contains a NUL byte or
/// is longer than 255 bytes. (The size restriction exists so that searching
/// for a label during deserialization can be done in small bounded space.)
pub(crate) fn write_label_len(label: &str) -> usize {
assert!(label.len() <= 255, "label must not be longer than 255 bytes");
assert!(label.bytes().all(|b| b != 0), "label must not contain NUL bytes");
let label_len = label.len() + 1; // +1 for the NUL terminator
label_len + padding_len(label_len)
}
/// Reads the endianness check from the beginning of the given slice and
/// confirms that the endianness of the serialized object matches the expected
/// endianness. If the slice is too small or if the endianness check fails,
/// this returns an error.
///
/// Upon success, the total number of bytes read is returned.
pub(crate) fn read_endianness_check(
slice: &[u8],
) -> Result<usize, DeserializeError> {
let (n, nr) = try_read_u32(slice, "endianness check")?;
assert_eq!(nr, write_endianness_check_len());
if n != 0xFEFF {
return Err(DeserializeError::endian_mismatch(0xFEFF, n));
}
Ok(nr)
}
/// Writes 0xFEFF as an integer using the given endianness.
///
/// This is useful for writing into the header of a serialized object. It can
/// be read during deserialization as a sanity check to ensure the proper
/// endianness is used.
///
/// Upon success, the total number of bytes written is returned.
pub(crate) fn write_endianness_check<E: Endian>(
dst: &mut [u8],
) -> Result<usize, SerializeError> {
let nwrite = write_endianness_check_len();
if dst.len() < nwrite {
return Err(SerializeError::buffer_too_small("endianness check"));
}
E::write_u32(0xFEFF, dst);
Ok(nwrite)
}
/// Returns the number of bytes written by the endianness check.
pub(crate) fn write_endianness_check_len() -> usize {
size_of::<u32>()
}
/// Reads a version number from the beginning of the given slice and confirms
/// that is matches the expected version number given. If the slice is too
/// small or if the version numbers aren't equivalent, this returns an error.
///
/// Upon success, the total number of bytes read is returned.
///
/// N.B. Currently, we require that the version number is exactly equivalent.
/// In the future, if we bump the version number without a semver bump, then
/// we'll need to relax this a bit and support older versions.
pub(crate) fn read_version(
slice: &[u8],
expected_version: u32,
) -> Result<usize, DeserializeError> {
let (n, nr) = try_read_u32(slice, "version")?;
assert_eq!(nr, write_version_len());
if n != expected_version {
return Err(DeserializeError::version_mismatch(expected_version, n));
}
Ok(nr)
}
/// Writes the given version number to the beginning of the given slice.
///
/// This is useful for writing into the header of a serialized object. It can
/// be read during deserialization as a sanity check to ensure that the library
/// code supports the format of the serialized object.
///
/// Upon success, the total number of bytes written is returned.
pub(crate) fn write_version<E: Endian>(
version: u32,
dst: &mut [u8],
) -> Result<usize, SerializeError> {
let nwrite = write_version_len();
if dst.len() < nwrite {
return Err(SerializeError::buffer_too_small("version number"));
}
E::write_u32(version, dst);
Ok(nwrite)
}
/// Returns the number of bytes written by writing the version number.
pub(crate) fn write_version_len() -> usize {
size_of::<u32>()
}
/// Reads a pattern ID from the given slice. If the slice has insufficient
/// length, then this panics. If the deserialized integer exceeds the pattern
/// ID limit for the current target, then this returns an error.
///
/// Upon success, this also returns the number of bytes read.
pub(crate) fn read_pattern_id(
slice: &[u8],
what: &'static str,
) -> Result<(PatternID, usize), DeserializeError> {
let bytes: [u8; PatternID::SIZE] =
slice[..PatternID::SIZE].try_into().unwrap();
let pid = PatternID::from_ne_bytes(bytes)
.map_err(|err| DeserializeError::pattern_id_error(err, what))?;
Ok((pid, PatternID::SIZE))
}
/// Reads a pattern ID from the given slice. If the slice has insufficient
/// length, then this panics. Otherwise, the deserialized integer is assumed
/// to be a valid pattern ID.
///
/// This also returns the number of bytes read.
pub(crate) fn read_pattern_id_unchecked(slice: &[u8]) -> (PatternID, usize) {
let pid = PatternID::from_ne_bytes_unchecked(
slice[..PatternID::SIZE].try_into().unwrap(),
);
(pid, PatternID::SIZE)
}
/// Write the given pattern ID to the beginning of the given slice of bytes
/// using the specified endianness. The given slice must have length at least
/// `PatternID::SIZE`, or else this panics. Upon success, the total number of
/// bytes written is returned.
pub(crate) fn write_pattern_id<E: Endian>(
pid: PatternID,
dst: &mut [u8],
) -> usize {
E::write_u32(pid.as_u32(), dst);
PatternID::SIZE
}
/// Attempts to read a state ID from the given slice. If the slice has an
/// insufficient number of bytes or if the state ID exceeds the limit for
/// the current target, then this returns an error.
///
/// Upon success, this also returns the number of bytes read.
pub(crate) fn try_read_state_id(
slice: &[u8],
what: &'static str,
) -> Result<(StateID, usize), DeserializeError> {
if slice.len() < StateID::SIZE {
return Err(DeserializeError::buffer_too_small(what));
}
read_state_id(slice, what)
}
/// Reads a state ID from the given slice. If the slice has insufficient
/// length, then this panics. If the deserialized integer exceeds the state ID
/// limit for the current target, then this returns an error.
///
/// Upon success, this also returns the number of bytes read.
pub(crate) fn read_state_id(
slice: &[u8],
what: &'static str,
) -> Result<(StateID, usize), DeserializeError> {
let bytes: [u8; StateID::SIZE] =
slice[..StateID::SIZE].try_into().unwrap();
let sid = StateID::from_ne_bytes(bytes)
.map_err(|err| DeserializeError::state_id_error(err, what))?;
Ok((sid, StateID::SIZE))
}
/// Reads a state ID from the given slice. If the slice has insufficient
/// length, then this panics. Otherwise, the deserialized integer is assumed
/// to be a valid state ID.
///
/// This also returns the number of bytes read.
pub(crate) fn read_state_id_unchecked(slice: &[u8]) -> (StateID, usize) {
let sid = StateID::from_ne_bytes_unchecked(
slice[..StateID::SIZE].try_into().unwrap(),
);
(sid, StateID::SIZE)
}
/// Write the given state ID to the beginning of the given slice of bytes
/// using the specified endianness. The given slice must have length at least
/// `StateID::SIZE`, or else this panics. Upon success, the total number of
/// bytes written is returned.
pub(crate) fn write_state_id<E: Endian>(
sid: StateID,
dst: &mut [u8],
) -> usize {
E::write_u32(sid.as_u32(), dst);
StateID::SIZE
}
/// Try to read a u16 as a usize from the beginning of the given slice in
/// native endian format. If the slice has fewer than 2 bytes or if the
/// deserialized number cannot be represented by usize, then this returns an
/// error. The error message will include the `what` description of what is
/// being deserialized, for better error messages. `what` should be a noun in
/// singular form.
///
/// Upon success, this also returns the number of bytes read.
pub(crate) fn try_read_u16_as_usize(
slice: &[u8],
what: &'static str,
) -> Result<(usize, usize), DeserializeError> {
try_read_u16(slice, what).and_then(|(n, nr)| {
usize::try_from(n)
.map(|n| (n, nr))
.map_err(|_| DeserializeError::invalid_usize(what))
})
}
/// Try to read a u32 as a usize from the beginning of the given slice in
/// native endian format. If the slice has fewer than 4 bytes or if the
/// deserialized number cannot be represented by usize, then this returns an
/// error. The error message will include the `what` description of what is
/// being deserialized, for better error messages. `what` should be a noun in
/// singular form.
///
/// Upon success, this also returns the number of bytes read.
pub(crate) fn try_read_u32_as_usize(
slice: &[u8],
what: &'static str,
) -> Result<(usize, usize), DeserializeError> {
try_read_u32(slice, what).and_then(|(n, nr)| {
usize::try_from(n)
.map(|n| (n, nr))
.map_err(|_| DeserializeError::invalid_usize(what))
})
}
/// Try to read a u16 from the beginning of the given slice in native endian
/// format. If the slice has fewer than 2 bytes, then this returns an error.
/// The error message will include the `what` description of what is being
/// deserialized, for better error messages. `what` should be a noun in
/// singular form.
///
/// Upon success, this also returns the number of bytes read.
pub(crate) fn try_read_u16(
slice: &[u8],
what: &'static str,
) -> Result<(u16, usize), DeserializeError> {
check_slice_len(slice, size_of::<u16>(), what)?;
Ok((read_u16(slice), size_of::<u16>()))
}
/// Try to read a u32 from the beginning of the given slice in native endian
/// format. If the slice has fewer than 4 bytes, then this returns an error.
/// The error message will include the `what` description of what is being
/// deserialized, for better error messages. `what` should be a noun in
/// singular form.
///
/// Upon success, this also returns the number of bytes read.
pub(crate) fn try_read_u32(
slice: &[u8],
what: &'static str,
) -> Result<(u32, usize), DeserializeError> {
check_slice_len(slice, size_of::<u32>(), what)?;
Ok((read_u32(slice), size_of::<u32>()))
}
/// Try to read a u128 from the beginning of the given slice in native endian
/// format. If the slice has fewer than 16 bytes, then this returns an error.
/// The error message will include the `what` description of what is being
/// deserialized, for better error messages. `what` should be a noun in
/// singular form.
///
/// Upon success, this also returns the number of bytes read.
pub(crate) fn try_read_u128(
slice: &[u8],
what: &'static str,
) -> Result<(u128, usize), DeserializeError> {
check_slice_len(slice, size_of::<u128>(), what)?;
Ok((read_u128(slice), size_of::<u128>()))
}
/// Read a u16 from the beginning of the given slice in native endian format.
/// If the slice has fewer than 2 bytes, then this panics.
///
/// Marked as inline to speed up sparse searching which decodes integers from
/// its automaton at search time.
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn read_u16(slice: &[u8]) -> u16 {
let bytes: [u8; 2] = slice[..size_of::<u16>()].try_into().unwrap();
u16::from_ne_bytes(bytes)
}
/// Read a u32 from the beginning of the given slice in native endian format.
/// If the slice has fewer than 4 bytes, then this panics.
///
/// Marked as inline to speed up sparse searching which decodes integers from
/// its automaton at search time.
#[cfg_attr(feature = "perf-inline", inline(always))]
pub(crate) fn read_u32(slice: &[u8]) -> u32 {
let bytes: [u8; 4] = slice[..size_of::<u32>()].try_into().unwrap();
u32::from_ne_bytes(bytes)
}
/// Read a u128 from the beginning of the given slice in native endian format.
/// If the slice has fewer than 16 bytes, then this panics.
pub(crate) fn read_u128(slice: &[u8]) -> u128 {
let bytes: [u8; 16] = slice[..size_of::<u128>()].try_into().unwrap();
u128::from_ne_bytes(bytes)
}
/// Checks that the given slice has some minimal length. If it's smaller than
/// the bound given, then a "buffer too small" error is returned with `what`
/// describing what the buffer represents.
pub(crate) fn check_slice_len<T>(
slice: &[T],
at_least_len: usize,
what: &'static str,
) -> Result<(), DeserializeError> {
if slice.len() < at_least_len {
return Err(DeserializeError::buffer_too_small(what));
}
Ok(())
}
/// Multiply the given numbers, and on overflow, return an error that includes
/// 'what' in the error message.
///
/// This is useful when doing arithmetic with untrusted data.
pub(crate) fn mul(
a: usize,
b: usize,
what: &'static str,
) -> Result<usize, DeserializeError> {
match a.checked_mul(b) {
Some(c) => Ok(c),
None => Err(DeserializeError::arithmetic_overflow(what)),
}
}
/// Add the given numbers, and on overflow, return an error that includes
/// 'what' in the error message.
///
/// This is useful when doing arithmetic with untrusted data.
pub(crate) fn add(
a: usize,
b: usize,
what: &'static str,
) -> Result<usize, DeserializeError> {
match a.checked_add(b) {
Some(c) => Ok(c),
None => Err(DeserializeError::arithmetic_overflow(what)),
}
}
/// Shift `a` left by `b`, and on overflow, return an error that includes
/// 'what' in the error message.
///
/// This is useful when doing arithmetic with untrusted data.
pub(crate) fn shl(
a: usize,
b: usize,
what: &'static str,
) -> Result<usize, DeserializeError> {
let amount = u32::try_from(b)
.map_err(|_| DeserializeError::arithmetic_overflow(what))?;
match a.checked_shl(amount) {
Some(c) => Ok(c),
None => Err(DeserializeError::arithmetic_overflow(what)),
}
}
/// Returns the number of additional bytes required to add to the given length
/// in order to make the total length a multiple of 4. The return value is
/// always less than 4.
pub(crate) fn padding_len(non_padding_len: usize) -> usize {
(4 - (non_padding_len & 0b11)) & 0b11
}
/// A simple trait for writing code generic over endianness.
///
/// This is similar to what byteorder provides, but we only need a very small
/// subset.
pub(crate) trait Endian {
/// Writes a u16 to the given destination buffer in a particular
/// endianness. If the destination buffer has a length smaller than 2, then
/// this panics.
fn write_u16(n: u16, dst: &mut [u8]);
/// Writes a u32 to the given destination buffer in a particular
/// endianness. If the destination buffer has a length smaller than 4, then
/// this panics.
fn write_u32(n: u32, dst: &mut [u8]);
/// Writes a u128 to the given destination buffer in a particular
/// endianness. If the destination buffer has a length smaller than 16,
/// then this panics.
fn write_u128(n: u128, dst: &mut [u8]);
}
/// Little endian writing.
pub(crate) enum LE {}
/// Big endian writing.
pub(crate) enum BE {}
#[cfg(target_endian = "little")]
pub(crate) type NE = LE;
#[cfg(target_endian = "big")]
pub(crate) type NE = BE;
impl Endian for LE {
fn write_u16(n: u16, dst: &mut [u8]) {
dst[..2].copy_from_slice(&n.to_le_bytes());
}
fn write_u32(n: u32, dst: &mut [u8]) {
dst[..4].copy_from_slice(&n.to_le_bytes());
}
fn write_u128(n: u128, dst: &mut [u8]) {
dst[..16].copy_from_slice(&n.to_le_bytes());
}
}
impl Endian for BE {
fn write_u16(n: u16, dst: &mut [u8]) {
dst[..2].copy_from_slice(&n.to_be_bytes());
}
fn write_u32(n: u32, dst: &mut [u8]) {
dst[..4].copy_from_slice(&n.to_be_bytes());
}
fn write_u128(n: u128, dst: &mut [u8]) {
dst[..16].copy_from_slice(&n.to_be_bytes());
}
}
#[cfg(all(test, feature = "alloc"))]
mod tests {
use super::*;
#[test]
fn labels() {
let mut buf = [0; 1024];
let nwrite = write_label("fooba", &mut buf).unwrap();
assert_eq!(nwrite, 8);
assert_eq!(&buf[..nwrite], b"fooba\x00\x00\x00");
let nread = read_label(&buf, "fooba").unwrap();
assert_eq!(nread, 8);
}
#[test]
#[should_panic]
fn bad_label_interior_nul() {
// interior NULs are not allowed
write_label("foo\x00bar", &mut [0; 1024]).unwrap();
}
#[test]
fn bad_label_almost_too_long() {
// ok
write_label(&"z".repeat(255), &mut [0; 1024]).unwrap();
}
#[test]
#[should_panic]
fn bad_label_too_long() {
// labels longer than 255 bytes are banned
write_label(&"z".repeat(256), &mut [0; 1024]).unwrap();
}
#[test]
fn padding() {
assert_eq!(0, padding_len(8));
assert_eq!(3, padding_len(9));
assert_eq!(2, padding_len(10));
assert_eq!(1, padding_len(11));
assert_eq!(0, padding_len(12));
assert_eq!(3, padding_len(13));
assert_eq!(2, padding_len(14));
assert_eq!(1, padding_len(15));
assert_eq!(0, padding_len(16));
}
}