Building Pluggable Python Applications

By Frank Wickström

The source for this blog post can be found on Github at andersinno/python_plugin_example.

Creating an application that can be extendable but have low coupling throughout is something that has always interested me. What if I could remove and add new parts to my application without thinking about the rest of the application breaking. Of course, this is often through as a programming 101 type of thing, “low coupling, high cohesion”. But saying the words versus actually being pragmatic is often challenging. This post will show one way you can keep your core application reasonably small and add new functionality using plugins.

One of the main advantages of using plugins is that they can be developed separately from the main application and be added or removed without the core of the application needing to think too much about them. This also means that plugins can be tested separately from the main application. It also means developers can add significant features to an application quickly without merging them into the main application. However, the main drawback is that plugin creators needs follow the constraints imposed by the core system and how the plugins are called.

In this post, we will create a toy example of how a pluggable application works using the pluggy library. The library is created by the pytest team, and you probably have used its functionality already when you have added extensions to PyTest such as coverage support.

How Pluggy works

How pluggy works is by first specifying your plugin specification and letting plugins use that specification to create extensions to your application. The plugins are called when the main applications call plugin hooks which correspond to the functions defined in the plugin specification. An example would be a process_file hook that the main application calls when a file should be processed for information. The application then passes the file as a parameter to the hook, and the plugin picks up the file and does whatever the plugin is written to do.

To know which plugins to call, Pluggy has a plugin manager that keeps track of all of the plugins available. There are two ways to register the plugins in an application. Either it is done manually by calling a register call and pointing to the plugin, or it is done automatically through setup.py entrypoint. In this post, we only go through the manual registration process, but you can read more about automating the registration from Pluggys documentation.

Pluggy has many features such as setting calling orders, blocking plugins, calling hooks retroactively if they are loaded later during runtime, etc. I mention these to point out that the library is very mature and if you don’t have a good reason for it, you probably are better off using Pluggy than rolling your own plugin manager. However, this post will keep it simple and show how to build a small toy application using plugins.

Building a pluggable application

We will write a tool that will collect jokes from plugins and display them on screen. The main application itself will not be responsible for getting any jokes; however, that will be the plugin’s job. To keep things simple, all plugins will be included in the main codebase, but they could just as well be written as separate Python packages as well.

Let’s start with the dependencies that will need to be installed in order to build this. Building this example application, Poetry is used as the package manager, but you can use what best fits you. The packages that we are going to need are:

requests  # This is needed for the plugins
pluggy

Next up, we create the base of our application:

# main.py
class App:
    def display_jokes(self, amount: int) -> None:
				results = [[]]
        jokes = [joke for plugin in results for joke in plugin]
        for joke in jokes:
            print(joke)
if __name__ == "__main__":
    app = App()
    app.display_jokes(2)

This code does almost nothing, and that is the point. We will leave the heavy lifting to the plugins in this case. But to get those plugins working, we first need to create our plugin specification. Let’s make that now.

# hookspec.py
from typing import Any, Callable, List, TypeVar, cast
import pluggy  # type: ignore
F = TypeVar("F", bound=Callable[..., Any])
hookspec = cast(Callable[[F], F], pluggy.HookspecMarker("python_plugin_example"))
class PluginExampleHookSpec:
    """
    Python Plugin Example Hook Specification
    """
    @hookspec
    def retrieve_joke(self, amount: int) -> List[str]:
        """
        Fired when retrieving jokes
        Args:
            amount: How many jokes should be returned
        Returns:
            A string containing a joke
        """

That’s it; without the typing and comment, this is five lines of code. The main lines of interest here are the hookspec assignment and the function decorator. The hookspec variable is created by calling the HookspecMarker with the name of the application as the parameter. So if your application is called “awesomefire” in your setup.py or pyproject.toml file, that is what you should enter as the marker parameter as well. Each of the hook functions you want to define should then be decorated with the @hookspec decorator to mark them as hooks. In this example, we have a single retrieve_joke function that takes one parameter amount that indicates how many results we would like to get back from the plugin. The documentation and typing, while optional, is highly encouraged as otherwise, plugin developers will have to guess what is being passed to them when the hook is called.

Now that we have our specifications, we also need to create an implementation function that plugins can hook into our specifications.

# hookimpl.py
from typing import Any, Callable, TypeVar, cast
import pluggy  # type: ignore
F = TypeVar("F", bound=Callable[..., Any])
hookimpl = cast(Callable[[F], F], pluggy.HookimplMarker("python_plugin_example"))

Plugins will later use the hookimpl variable to hook into the plugin system. Again, the name of the application is passed as the main parameter to the HookimplMarker. Now that we have our plugin system set up let's set up our application to use it.

# main.py
import pluggy  # type: ignore
from python_plugin_example.hookspec import PluginExampleHookSpec
class App:
    def __init__(self) -> None:
        self.pm: pluggy.PluginManager = pluggy.PluginManager("python_plugin_example")
        self.pm.add_hookspecs(PluginExampleHookSpec)
    def display_jokes(self, amount: int) -> None:
        results = self.pm.hook.retrieve_joke(amount=amount)
        jokes = [joke for plugin in results for joke in plugin]
        for joke in jokes:
            print(joke)
if __name__ == "__main__":
    app = App()
    app.display_jokes(2)

The two new lines added to the __init__ method create a plugin manager and registers the plugin hook specification. Again, the plugin manager parameter should be the same as the application name. This plugin manager can then be used to register plugins and call hooks. In the display_jokes function, we have also added a hook call to call all plugins and collect all of the jokes returned by them. Pluggy always returns plugin results as a list, and as the retrieve_joke joke function also returns a list, we flatten everything down to a single list and finally print all of the jokes one by one. Now let's create our plugins. We will make two of them, one returning dad jokes and the other Chuck Norris jokes. We will get all of the jokes through APIs to not need to come up with the jokes ourselves. The plugin code looks like this.

# plugins/icanhazdadjokes.py
from typing import List
import requests
from python_plugin_example.hookimpl import hookimpl
DAD_JOKE_API_ENDPOINT = "<https://icanhazdadjoke.com/>"
class ICanHazDadJokePlugin:
    @hookimpl
    def retrieve_joke(self, amount: int) -> List[str]:
        headers = {
            "Accept": "application/json",
        }
        values = []
        for i in range(amount):
            response = requests.get(DAD_JOKE_API_ENDPOINT, headers=headers)
            values.append(response.json().get("joke", ""))
        jokes = [f"👨 Dad joke: {value}" for value in values]
        return jokes
# plugins/chucknorris.py
from typing import List
import requests
from python_plugin_example.hookimpl import hookimpl
CHUCK_NORRIS_API_ENDPOINT = "<https://api.chucknorris.io/jokes/random>"
class ChuckNorrisPlugin:
    @hookimpl
    def retrieve_joke(self, amount: int) -> List[str]:
        values = []
        for i in range(amount):
            response = requests.get(CHUCK_NORRIS_API_ENDPOINT)
            values.append(response.json().get("value", ""))
        jokes = [f"🤠 Chuck Norris: {value}" for value in values]
        return jokes

The main parts to take note of here are the use of the @hookimpl decorator and the function name. The decorator marks a function as being something that can be called by a hook to distinguish it from other functions. The function retrieve_joke must match the hook specification that we created earlier and have the same attributes as it. If there is an attribute mismatch, Pluggy will complain, and your code will not work. The application code itself is relatively straightforward; we use the Request library to fetch jokes from APIs, turn the results into a list, and return the list. Now that we have our plugins written let's register those in our main application.

# main.py
import pluggy  # type: ignore
from python_plugin_example.hookspec import PluginExampleHookSpec
from python_plugin_example.plugins.chucknorris import ChuckNorrisPlugin
from python_plugin_example.plugins.icanhazdadjoke import ICanHazDadJokePlugin
class App:
    def __init__(self) -> None:
        self.pm: pluggy.PluginManager = pluggy.PluginManager("plugin_example")
        self.pm.add_hookspecs(PluginExampleHookSpec)
        self.pm.register(ChuckNorrisPlugin())
        self.pm.register(ICanHazDadJokePlugin())
    def display_jokes(self, amount: int) -> None:
        results = self.pm.hook.retrieve_joke(amount=amount)
        jokes = [joke for plugin in results for joke in plugin]
        for joke in jokes:
            print(joke)
if __name__ == "__main__":
    app = App()
    app.display_jokes(2)

The two new lines here are the self.pm.register calls that add the plugins to the plugin manager. Note that what is passed to the register are class instances and not the class itself.

And that is it. We now have an application that uses plugins to fetch jokes from the internet. The fetching logic of those jokes is fully decoupled from the main application code that only takes care of outputting the results in this case.

> python main.py
👨 Dad joke: Geology rocks, but Geography is where it’s at!
👨 Dad joke: My pet mouse ‘Elvis’ died last night. He was caught in a trap..
🤠 Chuck Norris: Chuck Norris only applies his car brakes for people with red hair & beards.
🤠 Chuck Norris: Chuck Norris does not need a toothbrush when he brushes his teeth

This is a simple example but could easily be extended to include more hooks and more plugins. So feel free to download the source for it and add your hooks and plugins. The source can be found on GitHub.