Unicode Slugs in Elixir: Persian, Hindi and Japanese URLs

Mishka Slug makes URL slugs that keep the script of the title. MishkaSlug.slugify/2 turns قیمت‌گذاری نرم‌افزار into قیمتگذاری-نرمافزار, a Hindi title into a Hindi slug and a Japanese title into a Japanese slug. This post shows how, with code you can paste into a terminal.

Mishka Slug is free and open source, like Mishka CMS that uses it. A founding seat pays for the time we spend on both, and on the extensions: you get lifetime access, priority support and every paid extension we release.

A lot of slug code makes ASCII only. A title in Persian, Hindi or Thai then gives an empty slug, or a slug with holes in it. Mishka CMS runs sites in many languages, so we wrote a package that treats every script the same way.

You will install it, slugify titles in eleven languages, tune the output with options, restrict a site to its own scripts and write the slug in an Ash resource. Each example below ran against Hex release 0.0.1.

Try it in one minute

You need Elixir 1.20 or newer, and nothing else. Mix.install fetches the package, so you can run this in a file or in iex without creating a project.

An iex session, step by step: three titles, then the two errors you can get.

Mix.install([{:mishka_slug, "0.0.1"}])

MishkaSlug.slugify("Hello World: Getting Started with Elixir!")
#=> {:ok, "hello-world-getting-started-with-elixir"}

MishkaSlug.slugify("قیمت‌گذاری نرم‌افزار")
#=> {:ok, "قیمتگذاری-نرمافزار"}

MishkaSlug.slugify!("Привет, мир!")
#=> "привет-мир"

MishkaSlug.slugify("😀🚀")
#=> {:error, :empty}

The function returns {:ok, slug} or {:error, reason}. slugify!/2 returns the slug or raises ArgumentError. Text with no letters or digits, like the two emoji above, is {:error, :empty}, so you never store an empty slug.

Eleven languages, one function

The same call works for every script, and it does not transliterate. These are real results from the release:

Look at the Hindi and Thai rows. Their vowel signs and tone marks are part of the word, so the slug keeps them. Arabic harakat and Hebrew points are optional spelling, so the slug drops them: שָׁלוֹם becomes שלום. The Persian half-space (ZWNJ) is removed, so قیمت‌گذاری becomes one word.

MishkaSlug.slugify("تعلّم البرمجة بسهولة")
#=> {:ok, "تعلم-البرمجة-بسهولة"}

MishkaSlug.slugify("हिन्दी भाषा नमस्ते")
#=> {:ok, "हिन्दी-भाषा-नमस्ते"}

MishkaSlug.slugify("สวัสดีชาวโลก")
#=> {:ok, "สวัสดีชาวโลก"}

MishkaSlug.slugify("東京キャブ乗り場 ラーメン")
#=> {:ok, "東京キャブ乗り場-ラーメン"}

MishkaSlug.slugify("你好，世界！学习 Elixir")
#=> {:ok, "你好-世界-学习-elixir"}

MishkaSlug.slugify("안녕하세요 세계")
#=> {:ok, "안녕하세요-세계"}

MishkaSlug.slugify("ΟΔΟΣ Αθηνών")
#=> {:ok, "οδος-αθηνών"}

Why not a regex?

A regex slugifier loses words or marks in most non-Latin scripts. We ran two common ones next to Mishka Slug on the same titles: [^a-z0-9]+ and [^\p{L}\p{N}]+, both replaced by -.

The same titles through three slugifiers. Output copied from a real run.

The ASCII regex gives an empty slug for every title that is not Latin. The letters regex is better, but \p{L} does not match combining marks. Hindi नमस्ते becomes नमस-त, and Thai สวัสดี becomes สว-สด. Here is the code that made the picture:

Mix.install([{:mishka_slug, "0.0.1"}])

# Two slug functions many projects write by hand.
ascii_only = fn t ->
  t |> String.downcase() |> String.replace(~r/[^a-z0-9]+/, "-") |> String.trim("-")
end

letters_only = fn t ->
  t |> String.downcase() |> String.replace(~r/[^\p{L}\p{N}]+/u, "-") |> String.trim("-")
end

titles = [
  {"English", "Hello World: Getting Started with Elixir!"},
  {"Persian", "قیمت‌گذاری نرم‌افزار"},
  {"Hindi", "हिन्दी भाषा नमस्ते"},
  {"Thai", "สวัสดีชาวโลก"},
  {"Hebrew", "שָׁלוֹם עוֹלָם"},
  {"Japanese", "東京キャブ乗り場 ラーメン"},
  {"Russian", "Привет, мир!"}
]

for {name, title} <- titles do
  {:ok, mishka} = MishkaSlug.slugify(title)
  IO.puts(name)
  IO.puts("  ascii regex   #{inspect(ascii_only.(title))}")
  IO.puts("  letters regex #{inspect(letters_only.(title))}")
  IO.puts("  MishkaSlug    #{inspect(mishka)}")
end

How a slug is made

A slug is lowercase words joined by -. A word is a run of letters and decimal digits, with the combining marks that complete them. Everything else, from spaces to emoji and / ? # % \ :, separates words. The docs page How a slug is made lists every rule.

The four stages of slugify/2, with a Persian and English example.

The result is normalized to NFC, so a stored slug equals a pasted one. Only Latin letters change: they lose accents the way WordPress writes them, so Café Crème brûlée gives cafe-creme-brulee. Greek, Cyrillic, Arabic and every other script keep their marks.

The docs page that lists every rule, in the 0.0.1 docs.

Tune it with options

Every option has a default that fits a typical site, and you change one only when you need to. The list below shows each choice with its real output. A misspelled option raises ArgumentError, so a typo never turns into a silently ignored setting.

# Persian half-space: glue the parts (default) or break the word
MishkaSlug.slugify("قیمت‌گذاری")                    #=> {:ok, "قیمتگذاری"}
MishkaSlug.slugify("قیمت‌گذاری", zwnj: :separator)  #=> {:ok, "قیمت-گذاری"}

# Digits: as typed (default) or 0-9
MishkaSlug.slugify("سال ۱۴۰۳")                      #=> {:ok, "سال-۱۴۰۳"}
MishkaSlug.slugify("سال ۱۴۰۳", digits: :ascii)      #=> {:ok, "سال-1403"}

# Arabic keyboard letters folded to Persian
MishkaSlug.slugify("كتاب يك")                       #=> {:ok, "كتاب-يك"}
MishkaSlug.slugify("كتاب يك", language: "fa")       #=> {:ok, "کتاب-یک"}

# Latin accents: removed (default) or kept
MishkaSlug.slugify("Café Crème")                    #=> {:ok, "cafe-creme"}
MishkaSlug.slugify("Café Crème", latin: :keep)      #=> {:ok, "café-crème"}
MishkaSlug.slugify("Straße")                        #=> {:ok, "strase"}
MishkaSlug.slugify("Straße", language: "de")        #=> {:ok, "strasse"}

# Characters kept inside a word
MishkaSlug.slugify("snake_case_name")               #=> {:ok, "snake_case_name"}
MishkaSlug.slugify("snake_case_name", keep: [])     #=> {:ok, "snake-case-name"}
MishkaSlug.slugify("Elixir v1.2 notes", keep: ["."]) #=> {:ok, "elixir-v1.2-notes"}

# Length: whole words while they fit
MishkaSlug.slugify("hello world foo", max_length: 13) #=> {:ok, "hello-world"}

The guide Control accents, separators and length has the details, and the options reference lists every default.

Allow only your site's languages

Pass locales: with the languages your site publishes in, and the slug keeps the scripts of those languages plus Latin. A letter of any other script is dropped. An empty list, the default, keeps every script.

MishkaSlug.slugify("آموزش React در ۱۰ دقیقه", locales: ["fa", "en"])
#=> {:ok, "آموزش-react-در-۱۰-دقیقه"}

MishkaSlug.slugify("قیمت Elixir", locales: ["en"])
#=> {:ok, "elixir"}

MishkaSlug.slugify("قیمت", locales: ["en"])
#=> {:error, :no_allowed_script}

MishkaSlug.slugify("hello", locales: ["xx"])
#=> {:error, {:unknown_locale, "xx"}}

Safe by default

A slug ends up in a URL, a file name and sometimes an HTML page, so Mishka Slug removes what a visitor cannot see. Zero-width characters, bidi controls such as U+202E, soft hyphens, controls and unassigned code points never reach the slug. A slug can never hold a URL delimiter, and it is never . or ...

MishkaSlug.slugify("pay\u{200B}pal")               #=> {:ok, "paypal"}
MishkaSlug.slugify("report\u{202E}fdp.exe")       #=> {:ok, "reportfdp-exe"}
MishkaSlug.slugify("a/b?c#d%e\\f:g")              #=> {:ok, "a-b-c-d-e-f-g"}
MishkaSlug.slugify("..")                           #=> {:error, :empty}

# Cyrillic "а" inside a Latin word
MishkaSlug.slugify("p\u{0430}ypal")                      #=> {:ok, "p-а-ypal"}
MishkaSlug.slugify("p\u{0430}ypal", mixed_scripts: :reject) #=> {:error, :mixed_scripts}

The Cyrillic а in pаypal looks like a Latin one. By default the word is cut where the script changes, which makes the trick visible. With mixed_scripts: :reject the title is refused. Japanese, Korean and Chinese words that mix Han with kana, Hangul or Bopomofo are one family each, so they are never cut.

Use it in an Ash resource

With Ash in your project, MishkaSlug.Change writes the slug inside an action, so you never call slugify/2 by hand. It exists only when Ash is present. Here is a complete script that runs without a database, on Ash's in-memory data layer:

Mix.install(
  [{:mishka_slug, "0.0.1"}, {:ash, "~> 3.37"}],
  config: [ash: [default_string_length_count: :codepoints]]
)

Logger.configure(level: :warning)

defmodule Demo.Page do
  use Ash.Resource, domain: Demo.Pages, data_layer: Ash.DataLayer.Ets

  attributes do
    uuid_primary_key :id
    attribute :title, :string, allow_nil?: false, public?: true
    attribute :language, :string, public?: true

    attribute :slug, :string,
      public?: true,
      constraints: [max_length: 60]
  end

  actions do
    defaults [:read]

    create :create do
      accept [:title, :language]

      change {MishkaSlug.Change,
              from: :title, into: :slug, language: :language, locales: ["fa", "en"]}
    end
  end
end

defmodule Demo.Pages do
  use Ash.Domain, validate_config_inclusion?: false

  resources do
    resource Demo.Page
  end
end

for {title, language} <- [
      {"قیمت‌گذاری نرم‌افزار", "fa"},
      {"Pricing your software", "en"},
      {"😀🚀", "en"}
    ] do
  Demo.Page
  |> Ash.Changeset.for_create(:create, %{title: title, language: language})
  |> Ash.create()
  |> case do
    {:ok, page} -> IO.puts("ok    #{inspect(page.slug)}")
    {:error, %{errors: [error | _]}} -> IO.puts("error on #{error.field}: #{error.message}")
  end
end

Running it prints:

ok    "قیمتگذاری-نرمافزار"
ok    "pricing-your-software"
error on title: has no letters or digits to make a slug from

A title that gives no slug is a changeset error on :title, and the slug attribute is never set to nil or an empty string. The error texts go through Gettext in the locale of the calling process.

The Ash guide: the change, a live preview in a form and unique slugs.

Serve the URL, and redirect the variants

slugify/2 is idempotent: a slug made with some options gives the same slug when you run it again with the same options. That makes a cheap canonical check. Slugify the path parameter, and redirect when the result differs. The visitor who pasted a slug typed on an Arabic keyboard lands on the one address.

defmodule Canonical do
  @opts [language: "fa", locales: ["fa", "en"]]

  # What to do with the slug a visitor asked for.
  def check(param) do
    case MishkaSlug.slugify(param, @opts) do
      {:ok, ^param} -> :ok
      {:ok, slug} -> {:redirect, slug}
      {:error, _reason} -> :not_found
    end
  end
end

Canonical.check("كتاب-يك")
#=> {:redirect, "کتاب-یک"}

Canonical.check("Elixir-Book")
#=> {:redirect, "elixir-book"}

Canonical.check("...")
#=> :not_found

On the wire a Persian slug is percent-encoded, for example %D9%82%DB%8C%D9%85%D8%AA..., and browsers show it as the written word. Your router hands your action the decoded text, so the function above compares the same strings you store.

Where it runs today

Mishka CMS core 0.0.5, which is a pre-release on Hex now, makes the names and slugs of its records with Mishka Slug, and Mishka Blog depends on it. A Persian site keeps Persian slugs, and an internal name that becomes a module or a file keeps Latin letters and digits. You can read the story of the CMS in Introducing Mishka CMS and the latest release in Mishka CMS 0.0.2.

Docs and links

The documentation lives at mishka.tools/slug/docs. These are the pages to start with:

The package is on Hex and the code is on GitHub. It is part of Mishka CMS, and it works in any Elixir project.

Support the work

A small team builds Mishka Slug and Mishka CMS. If you want to help us work on them full time, a founding seat gives you lifetime access, priority support and every paid extension we release.

You can also sponsor our open source work, or hire our team for a Phoenix LiveView project. The founding seat is explained in Mishka CMS Lifetime Access for $250.

Questions people ask about Unicode slugs in Elixir

Thank you

Mishka Slug stands on Elixir, Ash Framework and our own guarded_struct for its checked options. Thank you to everyone who builds them, and to you for reading this far.