How to look for unittest.TestCase subclasses defined in random Python scripts and run them in one shot

To ensure robustness of our Python application, members of the team had written unit test cases to assert that the various code units of the application are working as intended. Unit test cases that are meant to test the same program component are placed together as functions of the same class and contained in a Python script.

As the Python application grows, the number of test scripts grew as well. In an attempt to perform automated unit testing with Jenkins, the first thing that I did was to write a Python script that will look for all the test cases in the application directory and run them at one go.

This post describes the Python script in detail.

Gathering the requirements for the script

I first started off with a brief outline of what the script should achieve:

  1. The script should look into some predetermined folder(s) for Python files that contain classes that extends from the unittest.TestCase class. In my case, I will want to look into some sibling folders of my Python script.
  2. The script should be able to run new test scripts without any modifications.
  3. If any of the test case fails, the script shall exit with error so as to make Jenkins send an email to the developers.

Building the Python script that looks for unittest.TestCase subclasses defined in random Python scripts and run the test cases within the classes in one shot

With the requirements in mind, I went ahead to build the Python script. There are two main elements in the Python script:

  1. The Python class that is responsible for finding the unit tests that are located in directories that are siblings to the Python script.
  2. The code execution block that will run the unit tests that the Python class had gathered with an instance of unittest.TextTestRunner.

Defining the Python class that is responsible for finding the unit tests that are located in directories that are siblings to the Python script

I first define a Python class that I can use for finding test cases inside some specific directories.

import importlib
import os
import re
import unittest

class UnitTestFinder:
    
    # Stores the list of directory paths that unittest.TestCase subclasses
    # would reside in
    _directoryPathList = []
    
    # Stores the subclasses of unittest.TestCase 
    _unitTestsList = []
    
    # Defines a regular expression that whitelist files that may contain unittest.TestCase
    _testFilePathPattern = re.compile('.*[Tt]est.*\.py$')
    
    def addLookupDirectoryPath(self, directoryPath):
        self._directoryPathList.append(directoryPath)
    
    def gatherUnitTests(self):
        
        # For each indicated directory path
        for directoryPath in self._directoryPathList:
            
            if os.path.isdir(directoryPath):
                # Walk through the contents 
                for subFolderRoot, foldersWithinSubFolder, files in os.walk(directoryPath):
                
                   # look at the files for subclasses of unittest.TestCase
                   for file in files:
                       
                       fileBasename = os.path.basename(file)
                       if self._testFilePathPattern.match(fileBasename) :
        
                           # Import the relevant module (note: a module does not end with .py)
                           moduleDirectory = os.path.join(subFolderRoot, os.path.splitext(file)[0])
                           moduleStr = moduleDirectory.replace(os.path.sep, '.')
                           module = importlib.import_module(moduleStr)
        
                           # Look for classes that implements unittest.TestCase
                           for name in dir(module) :
                               moduleAttribute = getattr(module, name)
                               if isinstance(moduleAttribute, type) and issubclass(moduleAttribute, unittest.TestCase):
                                   
                                   # Use testLoader to load the tests that are defined in the unittest.TestCase class
                                   self._unitTestsList.append(unittest.defaultTestLoader.loadTestsFromTestCase(moduleAttribute))

            
    def getUnitTestsList(self) : 
        return self._unitTestsList

There are three functions in UnitTestFinder:

  1. The addLookupDirectoryPath function.
  2. The gatherUnitTests function.
  3. The getUnitTestsList function.
What the addLookupDirectoryPath function does

The addLookupDirectoryPath function remembers the specific sibling directories that we want our UnitTestFinder to look into. The path to these directories are stored as list items via the _directoryPathList variable.

What the gatherUnitTests function does

The gatherUnitTests function then loop through the directory paths and begin searching for definition of subclasses of unittest.TestCase inside each of the sibling directories.

Using the os.walk function on each of the directory path that we added via the addLookupDirectoryPath function, we check for files that start with the word 'test' or 'Test' and end with the '.py' extension via regular expression.

For Python files that match our regular expression, we use the importlib.import_module() function to import the Python file and make it accessible via the module variable.

We then loop through the attribute names in the module via the dir() function. With the getattr() function, we check for subclasses of the unittest.TestCase class.

If there are subclasses of unittest.TestCase, we use unittest.defaultTestLoader to help us load the test cases within the subclasses. We then append whatever unittest.defaultTestLoader returns us to self._unitTestsList.

What the getUnitTestsList function does

The getUnitTestsList functions returns the unit tests that the gatherUnitTests function finds.

Building the code execution block that will run the unit tests that the Python class had gathered with an instance of unittest.TextTestRunner

With UnitTestFinder defined , I then proceed to writing the code block that will utilise UnitTestFinder and run the test cases in one shot:

if __name__ == "__main__" :

    unitTestFinder = UnitTestFinder()
    unitTestFinder.addLookupDirectoryPath('folderWithTestCases')  
    unitTestFinder.gatherUnitTests()
    
    # Execute the test cases that had been collected by UnitTestFinder
    testSuite = unittest.TestSuite(unitTestFinder.getUnitTestsList())
    testResult = unittest.TextTestRunner(verbosity=2).run(testSuite) 

    # Throw an error to notify error condition.
    if not testResult.wasSuccessful() :
        raise ValueError('There were test failures.')

The code block starts with a test on whether the Python binary had ran our script first. If the script was ran first, we then:

  1. Create an instance of UnitTestFinder.
  2. Specify the directory to look for test cases, which in our case is just the directory 'folderWithTestCases'.
  3. Get the instance of UnitTestFinder to gather the test cases that are located inside of 'folderWithTestCases'.
  4. Create a unittest.TestSuite instance from the test cases that UnitTestFinder could gather and make it available via the testSuite variable.
  5. Get the test results by creating an instance of unittest.TextTestRunner and using it to run our test suite. We capture the result with the testResult variable.
  6. Finally we check whether the test run was successful via a call to testResult.wasSuccessful(). If not, we throw an instance of ValueError to get Jenkins to send an email to all the developers in the team.

Putting everything together

Putting the codes together, we will yield the following script:

import importlib
import os
import re
import unittest

class UnitTestFinder:
    
    # Stores the list of directory paths that unittest.TestCase subclasses
    # would reside in
    _directoryPathList = []
    
    # Stores the subclasses of unittest.TestCase 
    _unitTestsList = []
    
    # Defines a regular expression that whitelist files that may contain unittest.TestCase
    _testFilePathPattern = re.compile('.*[Tt]est.*\.py$')
    
    def addLookupDirectoryPath(self, directoryPath):
        self._directoryPathList.append(directoryPath)
    
    def gatherUnitTests(self):
        
        # For each indicated directory path
        for directoryPath in self._directoryPathList:
            
            if os.path.isdir(directoryPath):
                # Walk through the contents 
                for subFolderRoot, foldersWithinSubFolder, files in os.walk(directoryPath):
                
                   # look at the files for subclasses of unittest.TestCase
                   for file in files:
                       
                       fileBasename = os.path.basename(file)
                       if self._testFilePathPattern.match(fileBasename) :
        
                           # Import the relevant module (note: a module does not end with .py)
                           moduleDirectory = os.path.join(subFolderRoot, os.path.splitext(file)[0])
                           moduleStr = moduleDirectory.replace(os.path.sep, '.')
                           module = importlib.import_module(moduleStr)
        
                           # Look for classes that implements unittest.TestCase
                           for name in dir(module) :
                               moduleAttribute = getattr(module, name)
                               if isinstance(moduleAttribute, type) and issubclass(moduleAttribute, unittest.TestCase):
                                   
                                   # Use testLoader to load the tests that are defined in the unittest.TestCase class
                                   self._unitTestsList.append(unittest.defaultTestLoader.loadTestsFromTestCase(moduleAttribute))

            
    def getUnitTestsList(self) : 
        return self._unitTestsList

if __name__ == "__main__" :

    unitTestFinder = UnitTestFinder()
    unitTestFinder.addLookupDirectoryPath('folderWithTestCases')  
    unitTestFinder.gatherUnitTests()
    
    # Execute the test cases that had been collected by UnitTestFinder
    testSuite = unittest.TestSuite(unitTestFinder.getUnitTestsList())
    testResult = unittest.TextTestRunner(verbosity=2).run(testSuite) 

    # Throw an error to notify error condition.
    if not testResult.wasSuccessful() :
        raise ValueError('There were test failures.')
Trying out the script

To try out the script, I created a sample Python script, 'test_arithmetic_operations_a.py', and place it inside the directory, 'folderWithTestCases':

import unittest

class ArithmeticTestCases(unittest.TestCase) :

    def test_addition(self):
        self.assertEqual(1+1, 2)
        
    def test_multiplication(self):
        self.assertEqual(2*3, 6)

And another Python script, 'test_arithmetic_operations_b.py' and place it inside the directory, 'folderWithTestCases/anotherFolder':

import unittest

class ArithmeticTestCasesB(unittest.TestCase) :

    def test_addition(self):
        self.assertEqual(1+1, 2)
        
    def test_multiplication(self):
        self.assertEqual(2*3, 6)

I then run the following command inside of the directory that contains the Python script that I had written earlier:

python3 run_unit_tests_in_one_shot.py

Which produced the following output:

test_addition (folderWithTestCases.test_arithmetic_operations_a.ArithmeticTestCasesA) ... ok
test_multiplication (folderWithTestCases.test_arithmetic_operations_a.ArithmeticTestCasesA) ... ok
test_division (folderWithTestCases.anotherFolder.test_arithmetic_operations_b.ArithmeticTestCasesB) ... ok
test_subtraction (folderWithTestCases.anotherFolder.test_arithmetic_operations_b.ArithmeticTestCasesB) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.001s

OK

I then change the test_addition test case by asserting that 1+1 equals 3 in order to create a test failure:

import unittest

class ArithmeticTestCasesA(unittest.TestCase) :

    def test_addition(self):
        self.assertEqual(1+1, 3)
        
    def test_multiplication(self):
        self.assertEqual(2*3, 6)

This time, I got the following output:

test_addition (folderWithTestCases.test_arithmetic_operations_a.ArithmeticTestCasesA) ... FAIL
test_multiplication (folderWithTestCases.test_arithmetic_operations_a.ArithmeticTestCasesA) ... ok
test_division (folderWithTestCases.anotherFolder.test_arithmetic_operations_b.ArithmeticTestCasesB) ... ok
test_subtraction (folderWithTestCases.anotherFolder.test_arithmetic_operations_b.ArithmeticTestCasesB) ... ok

======================================================================
FAIL: test_addition (folderWithTestCases.test_arithmetic_operations_a.ArithmeticTestCasesA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/techcoil/poc/FindAndRunUnitTestsAtOneGoProject/folderWithTestCases/test_arithmetic_operations_a.py", line 6, in test_addition
    self.assertEqual(1+1, 3)
AssertionError: 2 != 3

----------------------------------------------------------------------
Ran 4 tests in 0.001s

FAILED (failures=1)
Traceback (most recent call last):
  File "run_unit_tests_in_one_shot.py", line 65, in <module>
    raise ValueError('There were test failures.')    
ValueError: There were test failures.

About Clivant

Clivant a.k.a Chai Heng enjoys composing software and building systems to serve people. He owns techcoil.com and hopes that whatever he had written and built so far had benefited people. All views expressed belongs to him and are not representative of the company that he works/worked for.