How to look for classes defined in Python 3 files dynamically

There are times when we need to write Python codes that extends the functionality of others, especially when we are writing a framework that allows for custom extension. To allow my Python program to be extended by others via the template design pattern, the first exploration task that I did was to find out how to dynamically look for classes defined in arbitrary Python files.

This post documents a way to look for classes defined in Python 3 files dynamically.

Creating a sample scenario

Suppose we are going to build a mechanism that allows custom Python codes to be notified of various events that our core Python 3 program generates while it is running. For the sake of simplicity, let's limit the scope of this exploration task to allowing custom codes to be notified after our core Python 3 program starts running.

Predefining the file structure of our Python program

In order not to exhaust our program too badly, we designate a folder to contain the custom codes. We name this folder as plugins. Codes that belong together should be contained in their own folder, for example, plugins/hello-world should contain the custom codes for us to output "Hello World!" when our program starts up.

The entry point which the plugin will be picked up by the core program should be done via a class that extends the Plugin class which is defined in the lib folder. To facilitate the management of plugins, let's have a PluginManager class which will help look up of classes that contain the custom codes so that the custom codes can be notified when events are generated from our core Python program.

With these rules in mind, we will have the following file structure:

program-root
  lib
    plugin.py
    plugin_manager.py
  plugin
    hello-world
      hello_world_plugin.py
  app.py

Defining the Plugin class

We first define the Plugin class which our custom codes will extend so that they can be notified when our core program generates some events:

import abc

class Plugin: 
    
    @abc.abstractmethod
    def event_triggered(self, event_name='', resource=''): 
        return

In the Plugin class, we defined an abstract method, event_triggered, that subclasses will need to override in order to be notified of events from the core program.

We put this class in program-root/lib/plugin.py.

Defining the HelloWorldPlugin class

We then define a sample subclass of the Plugin class, HelloWorldPlugin class:

from lib.plugin import Plugin

class HelloWorldPlugin(Plugin):
    def event_triggered(self, event_name, resource):
        print('Hello there! ', 'I got notified of: [', event_name, '] with the resource: [', resource, ']')

The HelloWorldPlugin implements the event_triggered method and prints out some text to standard output.

We put this class in program-root/plugins/hello-world/hello_world_plugin.py.

Defining the PluginManager class

The bulk of the logic of finding classes defined in Python 3 files lies in the PluginManager class:

import os
import importlib

from lib.plugin import Plugin

class PluginManager:
    
    __plugins_list = []
    
    def __init__(self):
        
        script_folder_path = os.path.dirname(os.path.realpath(__file__))
        plugin_folder_path = os.path.join(os.path.dirname(script_folder_path), 'plugins') 
        
        for subFolderRoot, foldersWithinSubFolder, files in os.walk(os.path.basename(plugin_folder_path)):
            
            for file in files:
                
                if os.path.basename(file).endswith('.py') : 
                    
                    # Import the relevant module (note: a module does not end with .py)
                    moduleDirectory = os.path.join(subFolderRoot, os.path.splitext(file)[0])
                    moduleStr = moduleDirectory.replace(os.path.sep, '.')
                    module = importlib.import_module(moduleStr)
                    
                    # Look for classes that implements Plugin class
                    for name in dir(module) :
                        obj = getattr(module, name)
                        if isinstance(obj, type) and issubclass(obj, Plugin) :
                            # Remember plugin
                            self.__plugins_list.append(obj())
                    
    def notify(self, event_name, resource = ''):
        for plugin in self.__plugins_list :
            plugin.event_triggered(event_name, resource);

The __plugins_list variable

The PluginManager class maintains a list of objects that extends from the Plugin class via the __plugins_list variable.

Inside the __init__ method

When the PluginManager class is instantiated via the __init__ method, PluginManager does the following:

1. Gets the directory where it resides in and make that information available via the script_folder_path variable.
2. Constructs the directory path of the plugins directory and make that information available via the plugin_folder_path variable.
3. Traverse the plugin directory and looks for Python files.

   When a Python file is detected, it will use the importlib.import_module() function to import the Python file and make it accessible via the module variable.

   It then loop through the attribute names in the module via the dir() function. With the getattr() function, it checks for subclasses of the Plugin class.

   If a subclass of the Plugin class is found, it constructs an object for each Plugin subclass and append it to __plugins_list.

Inside the notify method

The notify method allows the core program to notify the plugins when events are generated. When the notify method is executed, the event_triggered method of each Plugin subclass will be executed.

Defining the script which contains the core program

To demonstrate the concept of looking for classes defined in Python 3 files dynamically, we create a script, app.py, which the Python 3 executable will run:

import sys

from lib.plugin_manager import PluginManager

sys.path.append('.')

if(__name__ == '__main__') :
    
    pluginManager = PluginManager()
    pluginManager.notify('programStarted', 'We had gotten the program started')

When the script starts, an instance of PluginManger will be created. We then execute the notify() method to create the event when the core program had started.

Running app.py

With all the codes ready, we run the app.py with the Python 3 executable in our terminal shell:

python3 app.py

Doing this will result in the following output:

Hello there!  I got notified of: [ programStarted ] with the resource: [ We had gotten the program started ]

• Share
• Tweet
• Share
• Save