{"id":463,"date":"2016-03-27T18:10:44","date_gmt":"2016-03-27T10:10:44","guid":{"rendered":"https:\/\/www.techcoil.com\/blog\/?p=463"},"modified":"2018-09-04T22:58:51","modified_gmt":"2018-09-04T14:58:51","slug":"how-to-dynamically-look-for-classes-defined-in-python-3-file","status":"publish","type":"post","link":"https:\/\/www.techcoil.com\/blog\/how-to-dynamically-look-for-classes-defined-in-python-3-file\/","title":{"rendered":"How to look for classes defined in Python 3 files dynamically"},"content":{"rendered":"

There are times when we need to write Python codes that extends the functionality of others, especially when we are writing a framework that allows for custom extension. To allow my Python program to be extended by others via the template design pattern, the first exploration task that I did was to find out how to dynamically look for classes defined in arbitrary Python files. <\/p>\n

This post documents a way to look for classes defined in Python 3 files dynamically.<\/p>\n

Creating a sample scenario<\/h3>\n

Suppose we are going to build a mechanism that allows custom Python codes to be notified of various events that our core Python 3 program generates while it is running. For the sake of simplicity, let's limit the scope of this exploration task to allowing custom codes to be notified after our core Python 3 program starts running.<\/p>\n

Predefining the file structure of our Python program<\/h4>\n

In order not to exhaust our program too badly, we designate a folder to contain the custom codes. We name this folder as plugins<\/code>. Codes that belong together should be contained in their own folder, for example, plugins\/hello-world<\/code> should contain the custom codes for us to output \"Hello World!\" when our program starts up.<\/p>\n

The entry point which the plugin will be picked up by the core program should be done via a class that extends the Plugin<\/code> class which is defined in the lib<\/code> folder. To facilitate the management of plugins, let's have a PluginManager<\/code> class which will help look up of classes that contain the custom codes so that the custom codes can be notified when events are generated from our core Python program. <\/p>\n

With these rules in mind, we will have the following file structure:<\/p>\n

\r\nprogram-root\r\n  lib\r\n    plugin.py\r\n    plugin_manager.py\r\n  plugin\r\n    hello-world\r\n      hello_world_plugin.py\r\n  app.py\r\n<\/pre>\n
Defining the Plugin<\/code> class<\/h3>\n
We first define the Plugin<\/code> class which our custom codes will extend so that they can be notified when our core program generates some events:<\/p>\n
\r\nimport abc\r\n\r\nclass Plugin: \r\n    \r\n    @abc.abstractmethod\r\n    def event_triggered(self, event_name='', resource=''): \r\n        return\r\n\r\n<\/pre>\n
In the Plugin<\/code> class, we defined an abstract method, event_triggered<\/code>, that subclasses will need to override in order to be notified of events from the core program. <\/p>\n
We put this class in program-root\/lib\/plugin.py<\/code>.<\/p>\n
Defining the HelloWorldPlugin class<\/h3>\n
We then define a sample subclass of the Plugin<\/code> class, HelloWorldPlugin<\/code> class:<\/p>\n
\r\nfrom lib.plugin import Plugin\r\n\r\nclass HelloWorldPlugin(Plugin):\r\n    def event_triggered(self, event_name, resource):\r\n        print('Hello there! ', 'I got notified of: [', event_name, '] with the resource: [', resource, ']')\r\n<\/pre>\n
The HelloWorldPlugin<\/code> implements the event_triggered<\/code> method and prints out some text to standard output.<\/p>\n
We put this class in program-root\/plugins\/hello-world\/hello_world_plugin.py<\/code>.<\/p>\n
Defining the PluginManager class<\/h3>\n
The bulk of the logic of finding classes defined in Python 3 files lies in the PluginManager<\/code> class:<\/p>\n
\r\nimport os\r\nimport importlib\r\n\r\nfrom lib.plugin import Plugin\r\n\r\nclass PluginManager:\r\n    \r\n    __plugins_list = []\r\n    \r\n    def __init__(self):\r\n        \r\n        script_folder_path = os.path.dirname(os.path.realpath(__file__))\r\n        plugin_folder_path = os.path.join(os.path.dirname(script_folder_path), 'plugins') \r\n        \r\n        for subFolderRoot, foldersWithinSubFolder, files in os.walk(os.path.basename(plugin_folder_path)):\r\n            \r\n            for file in files:\r\n                \r\n                if os.path.basename(file).endswith('.py') : \r\n                    \r\n                    # Import the relevant module (note: a module does not end with .py)\r\n                    moduleDirectory = os.path.join(subFolderRoot, os.path.splitext(file)[0])\r\n                    moduleStr = moduleDirectory.replace(os.path.sep, '.')\r\n                    module = importlib.import_module(moduleStr)\r\n                    \r\n                    # Look for classes that implements Plugin class\r\n                    for name in dir(module) :\r\n                        obj = getattr(module, name)\r\n                        if isinstance(obj, type) and issubclass(obj, Plugin) :\r\n                            # Remember plugin\r\n                            self.__plugins_list.append(obj())\r\n                    \r\n    def notify(self, event_name, resource = ''):\r\n        for plugin in self.__plugins_list :\r\n            plugin.event_triggered(event_name, resource);\r\n<\/pre>\n
The __plugins_list<\/code> variable<\/h4>\n
The PluginManager<\/code> class maintains a list of objects that extends from the Plugin class via the __plugins_list<\/code> variable.<\/p>\n
Inside the __init__<\/code> method<\/h4>\n
When the PluginManager<\/code> class is instantiated via the __init__<\/code> method, PluginManager<\/code> does the following:<\/p>\n
    \n
  1. Gets the directory where it resides in<\/a> and make that information available via the script_folder_path<\/code> variable. <\/li>\n
  2. Constructs the directory path of the plugins directory and make that information available via the plugin_folder_path<\/code> variable.<\/li>\n
  3. Traverse the plugin directory<\/a> and looks for Python files.\n
    When a Python file is detected, it will use the importlib.import_module()<\/a><\/code> function to import the Python file and make it accessible via the module<\/code> variable.<\/p>\n
    It then loop through the attribute names in the module via the dir()<\/a><\/code> function. With the getattr()<\/a><\/code> function, it checks for subclasses of the Plugin<\/code> class. <\/p>\n
    If a subclass of the Plugin<\/code> class is found, it constructs an object for each Plugin<\/code> subclass and append it to __plugins_list<\/code>.\n<\/li>\n<\/ol>\n
    Inside the notify<\/code> method<\/h4>\n
    The notify<\/code> method allows the core program to notify the plugins when events are generated. When the notify<\/code> method is executed, the event_triggered<\/code> method of each Plugin<\/code> subclass will be executed.<\/p>\n
    Defining the script which contains the core program<\/h3>\n
    To demonstrate the concept of looking for classes defined in Python 3 files dynamically, we create a script, app.py<\/code>,  which the Python 3 executable will run:<\/p>\n
    \r\nimport sys\r\n\r\nfrom lib.plugin_manager import PluginManager\r\n\r\nsys.path.append('.')\r\n\r\nif(__name__ == '__main__') :\r\n    \r\n    pluginManager = PluginManager()\r\n    pluginManager.notify('programStarted', 'We had gotten the program started')\r\n\r\n<\/pre>\n
    When the script starts, an instance of PluginManger<\/code> will be created. We then execute the notify()<\/code> method to create the event when the core program had started.<\/p>\n
    Running app.py<\/h3>\n
    With all the codes ready, we run the app.py with the Python 3 executable in our terminal shell:<\/p>\n
    \r\npython3 app.py\r\n<\/pre>\n
    Doing this will result in the following output:<\/p>\n
    \r\nHello there!  I got notified of: [ programStarted ] with the resource: [ We had gotten the program started ]\r\n<\/pre>\n\n