Core Transforms¶

Generate a URL-safe slug from Unicode text.

Full pipeline: decode entities → transliterate → lowercase → strip non-alphanumeric → collapse separators → apply stopwords/max_length.

Shares python-slugify's core keyword parameters (separator, max_length, word_boundary, save_order, stopwords, lowercase, etc.), so slugify(text, ...) calls port directly. Note that disarm makes every parameter past text keyword-only, whereas python-slugify accepts some positionally.

Examples:

>>> slugify("Hello World!")
'hello-world'
>>> slugify("Straße nach München", lang="de")
'strasse-nach-muenchen'
>>> slugify("My Title", separator="_")
'my_title'
>>> slugify("The Big Fox", stopwords=["the"])
'big-fox'
>>> slugify("Very Long Title Here", max_length=10, word_boundary=True)
'very-long'
>>> slugify("🔥🔥🔥")
''
>>> slugify("🔥🔥🔥", default="n-a")
'n-a'
>>> slugify("🔥", default="N/A")  # default is sanitized, not returned raw
'n-a'

The output can be the empty string (#728).

Measured at Unicode 15.0.0, 243,399 single characters reduce to "" here (105,931 excluding the Private Use Area), and so does every string built from them. A caller keying a table on this has all of them, plus "no value", competing for one slot.

There is no on_empty here: this returns text rather than a key. The four key builders take one.

Unicode normalization.

Accepts a single string or a list of strings.

Examples:

>>> normalize("é", form="NFC")
'é'
>>> normalize(["é", "ño"], form="NFC")
['é', 'ño']

Replace Unicode confusable homoglyphs with target-script equivalents.

Uses Unicode TR39 confusables table. Characters without a confusable equivalent in the target script pass through unchanged (visual mapping only, not transliteration).

normalize_confusables("Москва", target_script="arabic")
# 'Мهсква'  — U+0647 ARABIC LETTER HEH replacing the Cyrillic о

normalize_confusables("ſ")   # 'f'  — TR39: a long s looks like an f
canonicalize("ſ")            # 's'  — NFKC decomposed it first

Examples:

>>> normalize_confusables("Ηello")  # Greek Η looks like Latin H
'Hello'
>>> normalize_confusables("раypal")  # Cyrillic р/а look like Latin p/a
'paypal'
>>> normalize_confusables("paypal", target_script="cyrillic")
'раураӏ'
>>> normalize_confusables("g००gle")  # Devanagari zeros stay numeric
'g00gle'
>>> normalize_confusables("२०२४", digit_policy="preserve")  # keep the script
'२०२४'
>>> normalize_confusables("g००gle", digit_policy="tr39")  # …or collide
'google'

Sanitize a string into a safe filename.

Transliterate → strip OS-illegal chars → collapse separators → handle reserved names (CON, NUL, etc.) → truncate respecting extension.

Examples:

>>> sanitize_filename("My Report (final).pdf")
'My_Report_(final).pdf'
>>> sanitize_filename("CON.txt")  # reserved on Windows
'_CON.txt'
>>> sanitize_filename("résumé.docx", lang="fr")
'resume.docx'

Remove diacritical marks while preserving base characters.

NFD decompose → strip combining marks → NFC recompose. Accepts a single string or a list of strings.

Destructive wherever a combining mark carries meaning, which is not only the Indic scripts (#624, #761). A Latin acute and a Devanagari vowel sign are both general category Mn, so both are removed — but in Latin an Mn is decoration and elsewhere it is part of the letter. José → Jose is readable. These are not::

বাংলা      → বল        Bengali, the vowel signs carry the word
हिन्दी      → हनद        Devanagari
မြန်မာ      → မနမ        Myanmar
かばん      → かはん      Japanese: the dakuten is the difference between
                         ば /ba/ and は /ha/, so this is a different word
Чайковский → Чаиковскии  Russian: й is a letter, not и with a mark; ё → е

Kana and Cyrillic are the two an "Indic scripts" warning sends a reader past. In kana the dakuten and handakuten are voicing, not decoration; in Cyrillic й and ё are letters of the alphabet that happen to decompose.

Use this for identifiers, filenames and search keys — where a deliberate many-to-one collapse is the point — and not for body text in any script whose marks are load-bearing. See Limitations (docs/limitations.md).

Examples:

>>> strip_accents("café résumé naïve")
'cafe resume naive'
>>> strip_accents(["café", "naïve"])
['cafe', 'naive']

Full Unicode case folding per CaseFolding.txt (Unicode 16.0).

Unlike str.lower(), this implements the complete Unicode Case Folding algorithm with all 1,557 status-C and status-F mappings. Covers Latin (ß→ss, ſ→s, İ→i̇), Greek (ς→σ, variant forms ϐ→β, ϑ→θ, ϕ→φ, ϖ→π, ϰ→κ, ϱ→ρ), Cyrillic, Armenian (ligature և→եւ), Georgian Mtavruli, Cherokee, Adlam, Deseret, Osage, Warang Citi, fullwidth Latin, and all Latin ligature expansions (fi→fi, fl→fl, ff→ff, ffi→ffi, ffl→ffl, ſt→st, st→st).

Equivalent to str.casefold() but executed in Rust via a compile-time PHF (perfect hash function) table. Pure-ASCII strings take a branchless fast path with no table lookup.

Examples:

>>> fold_case("Straße")
'strasse'
>>> fold_case("ΣΟΦΙΑ")
'σοφια'
>>> fold_case("find")
'find'

Fold all Unicode whitespace runs to single ASCII spaces, trimming the ends.

Folds whitespace only (#433): the line controls (TAB/LF/VT/FF/CR), the information separators (U+001C–U+001F), NEL, the Zs/Zl/Zp spaces, and the blank-rendering set (Braille blank, the Hangul fillers) each fold to a single space. It does not delete control or zero-width characters — for that, call strip_control_chars / strip_zero_width_chars, or use a preset that sequences them ahead of the fold (canonicalize and canonicalize_strict both do).

Folding the line controls (rather than deleting them) means a carriage return between two tokens becomes a space, never a silent join: "a\rb" → "a b".

Examples:

>>> collapse_whitespace("  hello   world  ")
'hello world'
>>> collapse_whitespace("tabs\there\ttoo")
'tabs here too'
>>> collapse_whitespace("a\rb")  # carriage return folds, not deletes
'a b'

Name every emoji, or replace every emoji with one string.

Two modes, and they read different tables because they answer different questions.

Naming (the default) asks what does CLDR call this?, so its domain is the CLDR name table — which is wider than the emoji: demojize("x™y") is "x trade mark y" because CLDR annotates U+2122.

Replacing (replacement=...) asks is this an emoji by the UCD's properties?, so its domain is the emoji-presentation set: Emoji_Presentation=Yes, an Emoji=Yes base carrying U+FE0F, and the ZWJ, modifier, keycap and flag sequences built on those. Nothing else moves — © and ™ stay, where naming would have written a word over them (#972).

Examples:

>>> demojize("I ❤️ Python 🐍")
'I red heart Python snake'
>>> demojize("aa🔥bb", replacement="")
'aabb'
>>> demojize("stop🛑now", replacement=" ")
'stop now'
>>> demojize("x©y", replacement="")
'x©y'

Replace every emoji with replacement, verbatim (#972).

The counterpart to demojize, and a different question of a different table. demojize asks what does CLDR call this?, so its domain is the CLDR name table, which is wider than the emoji: demojize("x™y") is "x trade mark y". This asks is this an emoji by the UCD's properties?, so its domain is the emoji-presentation set — Emoji_Presentation=Yes, an Emoji=Yes base carrying U+FE0F, and the ZWJ, modifier, keycap and flag sequences built on those. Nothing else moves.

Identical to demojize(text, replacement=...); this is the spelling every other binding carries, and the one to reach for when the operation is the point rather than a mode of naming.

Examples:

>>> replace_emoji("aa🔥bb")
'aabb'
>>> replace_emoji("stop🛑now", " ")
'stop now'
>>> replace_emoji("x©y")
'x©y'

Set a global emoji provider for all demojize calls.

The provider must implement the EmojiProvider protocol.

Pass None to reset to the built-in default (latest English CLDR).

Examples:

>>> set_emoji_provider(None)  # reset to default provider

Strip bidirectional override and formatting characters (UAX #9).

Removes: soft hyphen (U+00AD), Arabic Letter Mark (U+061C), LRM/RLM (U+200E/F), bidi embeddings/overrides (U+202A–U+202E), bidi isolates (U+2066–U+2069).

Keeps the logical order. This is a pure filter: the controls are deleted and the code-point order is untouched, so the result is the order the bytes are in, not the order a reader saw. "\u202e" + "paypal"[::-1] + "\u202c" renders as paypal and comes back as lapyap.

That is correct for a compiler, a filesystem or an identifier comparison, which all read logical order — the Trojan Source direction (CVE-2021-42574). It is the wrong answer for a search index, an NLP model or content moderation, which want what was displayed. disarm has no surface that returns display order; see "Stripping preserves logical order, not display order" in the limitations page (#740).

Examples:

>>> strip_bidi("hello\u200eworld")  # remove LRM
'helloworld'
>>> strip_bidi("hello\u061cworld")  # remove Arabic Letter Mark
'helloworld'
>>> strip_bidi("safe text")  # no bidi chars → unchanged
'safe text'

Fold typographic punctuation to its ASCII spelling (#703).

The dash family and the minus sign become -; the curly and low-9 quotes and the primes become ' / "; the ellipsis becomes ...; the non-standard spaces become a space. Nothing else in disarm does this as a stated purpose: canonicalize folds five dashes and skips the em dash and the horizontal bar, transliterate folds those two and rejects the other four, and a key built from either treats a—b and a-b as distinct while treating a–b and a-b as the same. A separate primitive rather than a change to either, because canonicalize is a security fold entitled to map “ to '' — a confusable skeleton for a double quote and a poor replacement for one.

Not covered, on purpose. U+3002 IDEOGRAPHIC FULL STOP and U+060C ARABIC COMMA are those scripts' own full stop and comma; the middle dot U+00B7 is a letter in Catalan l·l; the bullet stays. Spaces fold rather than delete, so words do not glue together.

Idempotent, and the identity on ASCII. Form-preserving, like the targeted strips: it folds one character class and composes nothing, so a decomposed letter leaves as it arrived. Compose it with a preset when boundary normalization is wanted too.

Examples:

>>> fold_punctuation("He said “ok” — then…")
'He said "ok" - then...'
>>> fold_punctuation("l·l")  # Catalan: a letter, not punctuation
'l·l'

Strip the Unicode Tags block (U+E0000–U+E007F) — the "ASCII smuggling" channel.

Preserves well-formed emoji subdivision flag sequences (U+1F3F4 + tag letters + U+E007F, e.g. the Scotland flag); stray tag characters (including the deprecated language tag U+E0001) are removed.

Examples:

>>> strip_tags("hi\U000e0050\U000e0057\U000e004e")  # tag-encoded "PWN"
'hi'

Strip every variation selector (VS1–VS16 and VS17–VS256).

These are the arbitrary-byte smuggling channel. Use strip_format if you need to keep the VS15/VS16 presentation selectors for rendering.

Examples:

>>> strip_variation_selectors("g\ufe01data")  # VS2
'gdata'

Strip every Unicode noncharacter (U+FDD0–U+FDEF, and U+xFFFE/U+xFFFF per plane).

Examples:

>>> strip_noncharacters("a\ufffeb")
'ab'

Strip every Private Use Area code point (BMP and planes 15/16).

PUA renders as arbitrary, font-defined glyphs (icon fonts, platform logos). Stripped by the comparison presets; use this helper to apply the same policy directly, or strip_format to preserve PUA for rendering.

Examples:

>>> strip_pua("a\ue000b")
'ab'

Strip excessive combining marks, preserving legitimate diacritics.

Caps the number of combining marks per base character at max_marks. Operates in NFD space and recomposes to NFC.

The default equals is_zalgo's threshold on purpose (#788). It was 2 while the threshold was 3, so this stripped from text the library had just declined to call suspicious: pointed and cantillated Hebrew routinely carries a vowel, a dot and an accent on one consonant, is_zalgo correctly returns False for it, and this removed the accent anyway. Three marks is ordinary text in Hebrew and Arabic; the Vietnamese ệ that set the original figure has two.

Examples:

>>> strip_zalgo("café")  # 1 combining mark — preserved
'café'
>>> strip_zalgo("Việt Nam")  # 2 marks — preserved
'Việt Nam'