How a slug is made

A slug is lowercase words joined by -. A word is a run of letters and decimal digits, with the combining marks that complete them, such as Hindi vowel signs and viramas or Thai tone marks. Everything else separates words: spaces, punctuation, emoji and the characters / ? # % \ :.

Three characters get special care. _ stays inside a word, as in WordPress. . and ~ separate words unless you list them in keep:. An apostrophe inside a word is removed. The result is normalized to NFC, so a stored slug compares equal to a pasted one.

Only Latin letters change. They lose their accents the way WordPress writes them: Café Crème gives cafe-creme, and Tiếng Việt gives tieng-viet. The language: option adds its own spelling, and latin: :keep leaves the accents.

Greek, Cyrillic, Arabic, Hebrew, Indic, Thai, Han, kana and Hangul keep their letters. Two details come from the script itself:

• Arabic harakat, tatweel and Hebrew points are optional spelling and are dropped. Every other mark is kept, because in Indic and Thai scripts the marks spell the word.
• A capital sigma that ends a Greek word becomes ς, in every language except tr and az.

MishkaSlug.slugify("שָׁלוֹם עוֹלָם")
#=> {:ok, "שלום-עולם"}

MishkaSlug.slugify("สวัสดีชาวโลก")
#=> {:ok, "สวัสดีชาวโลก"}

MishkaSlug.slugify("ΟΔΟΣ")
#=> {:ok, "οδος"}

A word never mixes scripts, so a look-alike letter cannot pass for a Latin letter inside a word. Japanese (Han with Hiragana and Katakana), Korean (Han with Hangul) and Chinese with Bopomofo are one family each, so their words are never cut. Characters that belong to no single script, such as ASCII digits, stay with the letters around them.

The locales: option adds an allow-list on top: only the scripts of the site's languages, and Latin, may appear. A digit follows the allow-list like a letter, but it never makes a word mixed, so iphone۱۵ stays one word.

MishkaSlug.slugify("iphone۱۵")
#=> {:ok, "iphone۱۵"}

These rules are always on and no option turns them off.

• Invisible characters are removed without leaving a break: zero-width spaces and joiners, bidi controls and isolates such as U+202E, soft hyphens, byte-order marks, variation selectors, control, private-use and unassigned code points.
• A slug never holds / ? # % \ : or whitespace, and is never . or ...
• The text is NFKC-normalized first, so fullwidth letters, ligatures and Arabic presentation forms become their plain letters.

MishkaSlug.slugify("pay" <> <<0x200B::utf8>> <> "pal")
#=> {:ok, "paypal"}

MishkaSlug.slugify("evil" <> <<0x202E::utf8>> <> "fdp.exe")
#=> {:ok, "evilfdp-exe"}

MishkaSlug.slugify("a/b?c#d%e\\f:g")
#=> {:ok, "a-b-c-d-e-f-g"}

MishkaSlug.slugify("..")
#=> {:error, :empty}

slugify/2 returns {:ok, slug} for every slug it made with the same options. A site can therefore slug an incoming URL parameter again and redirect when the result differs. That also folds the variants people paste: uppercase, the Arabic ي for the Persian ی, and decomposed accents.

check = fn param ->
  case MishkaSlug.slugify(param) do
    {:ok, ^param} -> :render
    {:ok, slug} -> {:redirect, slug}
    {:error, _reason} -> :not_found
  end
end

check.("hello-world")
#=> :render

check.("Hello-World")
#=> {:redirect, "hello-world"}

check.("😀")
#=> :not_found

• Uniqueness and reserved words stay the job of the resource that stores the slug.
• Do not use Mishka Slug for internal identifiers such as module names, CSS class names, file paths, atoms or database column names. Those need ASCII, and this library keeps non-Latin letters on purpose.

• Control accents, separators and length: the options that change the defaults.
• Keep the scripts your site allows: the allow-list in practice.
• Page fields: the path rules of a page in Mishka CMS.
• Mishka Blog and Mishka Document: extensions of Mishka CMS whose addresses use slugs.