How to look for classes defined in Python 3 files dynamically
There are times when we need to write Python codes that extends the functionality of others, especially when we are writing a framework that allows for custom extension. To allow my Python program to be extended by others via the template design pattern, the first exploration task that I did was to find out how to dynamically look for classes defined in arbitrary Python files.
This post documents a way to look for classes defined in Python 3 files dynamically.
Creating a sample scenario
Suppose we are going to build a mechanism that allows custom Python codes to be notified of various events that our core Python 3 program generates while it is running. For the sake of simplicity, let's limit the scope of this exploration task to allowing custom codes to be notified after our core Python 3 program starts running.
Predefining the file structure of our Python program
In order not to exhaust our program too badly, we designate a folder to contain the custom codes. We name this folder as
plugins. Codes that belong together should be contained in their own folder, for example,
plugins/hello-world should contain the custom codes for us to output "Hello World!" when our program starts up.
The entry point which the plugin will be picked up by the core program should be done via a class that extends the
Plugin class which is defined in the
lib folder. To facilitate the management of plugins, let's have a
PluginManager class which will help look up of classes that contain the custom codes so that the custom codes can be notified when events are generated from our core Python program.
With these rules in mind, we will have the following file structure:
program-root lib plugin.py plugin_manager.py plugin hello-world hello_world_plugin.py app.py
We first define the
Plugin class which our custom codes will extend so that they can be notified when our core program generates some events:
import abc class Plugin: @abc.abstractmethod def event_triggered(self, event_name='', resource=''): return
Plugin class, we defined an abstract method,
event_triggered, that subclasses will need to override in order to be notified of events from the core program.
We put this class in
Defining the HelloWorldPlugin class
We then define a sample subclass of the
from lib.plugin import Plugin class HelloWorldPlugin(Plugin): def event_triggered(self, event_name, resource): print('Hello there! ', 'I got notified of: [', event_name, '] with the resource: [', resource, ']')
HelloWorldPlugin implements the
event_triggered method and prints out some text to standard output.
We put this class in
Defining the PluginManager class
The bulk of the logic of finding classes defined in Python 3 files lies in the
import os import importlib from lib.plugin import Plugin class PluginManager: __plugins_list =  def __init__(self): script_folder_path = os.path.dirname(os.path.realpath(__file__)) plugin_folder_path = os.path.join(os.path.dirname(script_folder_path), 'plugins') for subFolderRoot, foldersWithinSubFolder, files in os.walk(os.path.basename(plugin_folder_path)): for file in files: if os.path.basename(file).endswith('.py') : # Import the relevant module (note: a module does not end with .py) moduleDirectory = os.path.join(subFolderRoot, os.path.splitext(file)) moduleStr = moduleDirectory.replace(os.path.sep, '.') module = importlib.import_module(moduleStr) # Look for classes that implements Plugin class for name in dir(module) : obj = getattr(module, name) if isinstance(obj, type) and issubclass(obj, Plugin) : # Remember plugin self.__plugins_list.append(obj()) def notify(self, event_name, resource = ''): for plugin in self.__plugins_list : plugin.event_triggered(event_name, resource);
PluginManager class maintains a list of objects that extends from the Plugin class via the
PluginManager class is instantiated via the
PluginManager does the following:
- Gets the directory where it resides in and make that information available via the
- Constructs the directory path of the plugins directory and make that information available via the
- Traverse the plugin directory and looks for Python files.
When a Python file is detected, it will use the
importlib.import_module()function to import the Python file and make it accessible via the
If a subclass of the
Pluginclass is found, it constructs an object for each
Pluginsubclass and append it to
notify method allows the core program to notify the plugins when events are generated. When the
notify method is executed, the
event_triggered method of each
Plugin subclass will be executed.
Defining the script which contains the core program
To demonstrate the concept of looking for classes defined in Python 3 files dynamically, we create a script,
app.py, which the Python 3 executable will run:
import sys from lib.plugin_manager import PluginManager sys.path.append('.') if(__name__ == '__main__') : pluginManager = PluginManager() pluginManager.notify('programStarted', 'We had gotten the program started')
When the script starts, an instance of
PluginManger will be created. We then execute the
notify() method to create the event when the core program had started.
With all the codes ready, we run the app.py with the Python 3 executable in our terminal shell:
Doing this will result in the following output:
Hello there! I got notified of: [ programStarted ] with the resource: [ We had gotten the program started ]