Ship and remove plugins with a runtime-installed library

• Read the documentation on installing a library at runtime for the install and uninstall calls, and the documentation on how events and plugins work for statuses, the compiled chain and the cannot_run answer.
• Call the functions on this page from trusted code, such as an admin task or a remote console. They write the plugin registry and the code path.
• The examples use a library named demo with one plugin, Demo.Greeter, on the event demo:greeting.

The library is an ordinary OTP application that depends on :mishka_installer. Its plugin is a module that calls use MishkaInstaller.Event.Hook, and the library's own application supervisor lists it, as in Write a plugin for an event:

defmodule Demo.Greeter do
  use MishkaInstaller.Event.Hook, event: "demo:greeting"

  @impl true
  def call(state), do: {:reply, Map.put(state, :greeting, "hello")}
end

defmodule Demo.Application do
  use Application

  @impl true
  def start(_type, _args) do
    Supervisor.start_link([Demo.Greeter], strategy: :one_for_one, name: Demo.Supervisor)
  end
end

In the library's mix.exs, application/0 names that module with mod: {Demo.Application, []}, and deps/0 lists {:mishka_installer, "~> 0.1.14"}. Mix then writes mod and the dependency into ebin/demo.app.

An install does nothing special for plugins. It adds the package's ebin to the code path, loads the application and the modules it lists, and calls Application.ensure_all_started/1. That runs Demo.Application.start/2, which starts Demo.Greeter. The plugin registers itself and starts like any plugin in a supervision tree. The boot of a node does the same for every stored library, so a plugin in an installed library needs no extra wiring. To add work to that start, such as opening a connection, override the plugin's start/0. The documentation on overriding plugin functions shows how.

Installer.install/1 returns when the application has started. The plugin registers and builds its event a moment later, in its own process, so a script that calls the event right after an install should wait for the plugin first. The documentation on testing plugins shows how to wait.

A plugin record keeps the application it belongs to in extension. The macro takes it from the app: of the Mix project that compiled the plugin, which for a library is its own mix.exs. Read the plugins of a library by that name:

alias MishkaInstaller.Event.Event

Enum.map(Event.get(:extension, :demo), &{&1.name, &1.event, &1.status})
# [{Demo.Greeter, "demo:greeting", :started}]

Event.get/2 takes the application as an atom, :demo. The install record and Installer.uninstall/1 use the string "demo".

Installing a higher version of an installed application is an upgrade. For the plugins it goes like this:

1. The installer puts the new ebin first on the code path.
2. It stops the running application, which stops the plugin processes, and unloads it.
3. It loads the new .app file, then loads each module that file lists and the node has not loaded yet.
4. It starts the application again. Each plugin process registers again. The record is kept with its status, and the priority, dependencies and extra data are applied again from the new code.

Step 3 skips every module the node already has. After an upgrade from 0.1.0 to 0.2.0 that both contain Demo.Greeter, the application reports version 0.2.0 while Demo.Greeter still runs the code of 0.1.0. A module that is new in 0.2.0 loads from the new package. The event is rebuilt only if a plugin's priority, dependencies or extra data changed.

Mishka Installer has no call that reloads the modules of an installed library. A restart does: the boot loads the library from the folder in its stored record, which is the new version, with nothing loaded before. Plan a restart after any upgrade that changes a plugin.

A plugin that the new version dropped keeps its record, and its old module stays loaded and in the chain. The library calls a plugin gone only when its application is loaded, lists no such module, and the module is not loaded. On an upgraded node the module is still loaded, so the check finds nothing until the restart. After the restart the boot logs unregistered Demo.Greeter from "demo:greeting": :demo no longer has this module and emits the [:mishka_installer, :event, :unregistered_gone] telemetry event. To remove it earlier, unregister it by hand as in the cleanup below.

Installer.uninstall/1 stops and unloads the application, removes its ebin from the code path, deletes the package folder and deletes the install record. It never calls the plugin registry and it never removes a loaded module. So after it:

• The plugin records stay in the registry with the status :started.
• The plugin modules stay loaded and in the compiled chain. The event keeps running them until the node restarts. Their processes are gone, since the application stopped.

The trouble starts at the next restart:

1. Mnesia brings the plugin records back, because they are stored on disc.
2. The boot finds no install record for the library, so it loads nothing and the plugin modules do not exist.
3. Event.unregister_gone/1 leaves the records. It removes a plugin only when its application is loaded and holds no such module. The uninstalled application is not loaded, and the library reads that as an extension that is still on its way in.
4. The event handler builds the event in :error mode, which runs no plugin, and logs "demo:greeting" cannot run: [Demo.Greeter] not loaded.

From then on, every call to that event answers its cannot_run: value, or the data it was given. The healthy plugins of the same event stop running too, since the event runs all of its plugins or none. A caller that passed no cannot_run: sees no error. Its data comes back unchanged.

Nothing raises. The signs are the error log line with cannot run and the health report. Before the restart, Hook.plugin_health(Demo.Greeter) shows alive?: false. After it, Hook.event_health("demo:greeting") shows mode: :error and healthy?: false.

Unregister the library's plugins first, then uninstall it. Event.unregister/3 deletes the plugin record, rebuilds the event without it, holds the plugins that depended on it, and stops the plugin's process with a normal exit, which its supervisor does not restart. The third argument is queue: false returns after the rebuild, and true queues it.

alias MishkaInstaller.Event.Event
alias MishkaInstaller.Installer.Installer

for plugin <- Event.get(:extension, :demo) do
  {:ok, _record} = Event.unregister(:name, plugin.name, false)
end

Installer.uninstall(%{app: "demo", version: "0.1.0"})
# :ok

Nothing is left in the registry, and the event keeps running the plugins of other applications. A restart finds nothing to clean.

Event.unregister/3 does not run an unregister/0 override of the plugin. To run the cleanup of such an override, call plugin.name.unregister() in the loop instead, while the module is loaded. The documentation on overriding plugin functions shows what the override does.

If the library was uninstalled without this step, the same loop repairs the registry. It needs no uninstall, and it works before the restart and after it:

for plugin <- Event.get(:extension, :demo) do
  Event.unregister(:name, plugin.name, true)
end

Each call deletes a record and rebuilds the event. The rebuild reads the registry again, so an event that was answering cannot_run: runs the plugins that are left. To list the leftovers without knowing the application, ask for the plugin records whose application is not loaded:

Enum.reject(Event.get(), &Application.spec(&1.extension, :vsn))

Check the result with Event.get(:extension, :demo), which returns [], and with a call to the event.

An install changes the node that runs it: its code path, its loaded modules and its started applications. The install record and the plugin records go to Mnesia and are copied to every node. So after an install on one node, the nodes disagree:

1. Node A runs the install. The library loads on A and its plugin registers.
2. A rebuilds the event and announces the rebuild with a :cluster_recompile message, which the PubSub messages documentation lists. Every other node rebuilds the event from the replicated registry.
3. Node B does not have the module. Its event is built in :error mode. It logs cannot run and answers cannot_run: or the data to every call until the module is loaded there.

Node B does not load the library on its own. The install record is replayed only when a node boots, and the record holds the absolute paths of the node that installed. A node whose disk does not have the folder at that path logs skipped re-loading demo on boot.

To roll a library out to a cluster:

1. Copy the package folder to every node, at the same path.
2. Run Installer.install/1 with the same map on every node that has not run it yet, one after the other. A node that finds the install record already stored updates it.
3. Rebuild the event once every node has the code. The rebuild is announced to the cluster, so one node is enough:

MishkaInstaller.Event.EventHandler.do_compile("demo:greeting", :re_event, false)

Once one node has installed, restarting the others after step 1 reaches the same state, because the boot replays the stored record from the folder. Between the first install and the last, the nodes without the code answer cannot_run:, so install on all nodes in one sitting. The deployment documentation covers how nodes join and rebuild.

An uninstall is node-local in the same way. Run it on every node that has the library. The plugin records are one set for the cluster, so unregister them once. Event.unregister/3 stops the plugin process only on the node where it runs.

This is the sequence on one node, with the demo library of the first section. In a console, give the plugin a moment to register after the install:

alias MishkaInstaller.Event.{Event, Hook}
alias MishkaInstaller.Installer.Installer

Installer.install(%{app: "demo", version: "0.1.0", path: "/data/extensions/demo-0.1.0"})
Hook.call("demo:greeting", %{})
# %{greeting: "hello"}

Installer.uninstall(%{app: "demo", version: "0.1.0"})
# :ok
Enum.map(Event.get(:extension, :demo), & &1.status)
# [:started]
Hook.call("demo:greeting", %{})
# %{greeting: "hello"}

The node restarts. The registry still lists the plugin and nothing loads the library:

Hook.call("demo:greeting", %{}, cannot_run: :refused)
# :refused
Hook.call("demo:greeting", %{})
# %{}

for plugin <- Event.get(:extension, :demo) do
  Event.unregister(:name, plugin.name, true)
end

Event.get(:extension, :demo)
# []
Hook.call("demo:greeting", %{})
# %{}

The last call answers its input because no plugin is left on the event. A plugin of another application on the same event would run again.

• Install a library at runtime: the install and uninstall calls.
• How the runtime installer works: what is stored and what the boot replays.
• Manage plugins while the app runs: the lifecycle calls.
• Event reference: unregister/3, unregister_gone/1 and EventHandler.do_compile/3.
• Deploy to production: a cluster, its seed node and its rebuilds.