Module text 

Text normalisation: case folding and diacritic folding with tables of our own, and the offset utilities needed to report ranges in the caller’s text.

Normalizer maps a &str to the form the index stores. Build the trie with Trie::build_normalized and search with Searcher::search_text so that terms and queries go through the same function.

What is folded, and what is not, is fixed in docs/design/unicode.md (section 2). In short, the default normaliser:

• drops diacritics of the letters of Latin-1 Supplement and Latin Extended-A (U+00C0 to U+017F), expands æ, œ, ß, ij, þ and the ligatures U+FB00 to U+FB06, and drops the combining marks U+0300 to U+036F;
• then folds case (Unicode full case folding) in ASCII and those blocks.

Everything else (other scripts, digits, punctuation, spaces, control characters, joiners, emoji) is copied unchanged. There is no locale: I always folds to i.

use keyhammer::text::Normalizer;

let n = Normalizer::new();
assert_eq!(n.normalize("São Paulo"), "sao paulo");
assert_eq!(n.normalize("Straße"), "strasse");
assert_eq!(n.normalize("Œuvre"), "oeuvre");
assert_eq!(n.normalize("final"), "final");
// Case only: accents are kept.
let case = Normalizer::new().with_diacritic_folding(false);
assert_eq!(case.normalize("CAFÉ"), "café");

Structs

Normalizer

Folds case and diacritics, identically for terms and queries.

SourceMap

Where each code point of a normalised string came from in its source.

Functions

utf8_offset

The byte offset in s of code point index cp: s.len() for cp == s.chars().count(), None beyond that.

utf8_range

A range of code points of s as a range of bytes; None if it is not within s or is reversed.

utf16_offset

The UTF-16 offset (in 16-bit units, as JavaScript counts) in s of code point index cp; None beyond the end.

utf16_range

A range of code points of s as a range of UTF-16 units; None if it is not within s or is reversed.