{"id":469,"date":"2016-04-03T12:45:30","date_gmt":"2016-04-03T04:45:30","guid":{"rendered":"https:\/\/www.techcoil.com\/blog\/?p=469"},"modified":"2018-09-04T22:59:27","modified_gmt":"2018-09-04T14:59:27","slug":"how-to-look-for-unittest-testcase-subclasses-defined-in-random-python-scripts-and-run-them-in-one-shot","status":"publish","type":"post","link":"https:\/\/www.techcoil.com\/blog\/how-to-look-for-unittest-testcase-subclasses-defined-in-random-python-scripts-and-run-them-in-one-shot\/","title":{"rendered":"How to look for unittest.TestCase subclasses defined in random Python scripts and run them in one shot"},"content":{"rendered":"

To ensure robustness of our Python application, members of the team had written unit test cases to assert that the various code units of the application are working as intended. Unit test cases that are meant to test the same program component are placed together as functions of the same class and contained in a Python script. <\/p>\n

As the Python application grows, the number of test scripts grew as well. In an attempt to perform automated unit testing with Jenkins, the first thing that I did was to write a Python script that will look for all the test cases in the application directory and run them at one go. <\/p>\n

This post describes the Python script in detail.<\/p>\n

Gathering the requirements for the script<\/h3>\n

I first started off with a brief outline of what the script should achieve:<\/p>\n

    \n
  1. The script should look into some predetermined folder(s) for Python files that contain classes<\/a> that extends from the unittest.TestCase<\/code><\/a> class. In my case, I will want to look into some sibling folders of my Python script.<\/li>\n
  2. The script should be able to run new test scripts without any modifications.<\/li>\n
  3. If any of the test case fails, the script shall exit with error so as to make Jenkins send an email to the developers.<\/li>\n<\/ol>\n

    Building the Python script that looks for unittest.TestCase<\/code> subclasses defined in random Python scripts and run the test cases within the classes in one shot<\/h3>\n

    With the requirements in mind, I went ahead to build the Python script. There are two main elements in the Python script:<\/p>\n

      \n
    1. The Python class that is responsible for finding the unit tests that are located in directories that are siblings to the Python script.<\/li>\n
    2. The code execution block that will run the unit tests that the Python class had gathered with an instance of unittest.TextTestRunner<\/code><\/a>.<\/li>\n<\/ol>\n

      Defining the Python class that is responsible for finding the unit tests that are located in directories that are siblings to the Python script<\/h4>\n

      I first define a Python class that I can use for finding test cases inside some specific directories.<\/p>\n

      \r\nimport importlib\r\nimport os\r\nimport re\r\nimport unittest\r\n\r\nclass UnitTestFinder:\r\n    \r\n    # Stores the list of directory paths that unittest.TestCase subclasses\r\n    # would reside in\r\n    _directoryPathList = []\r\n    \r\n    # Stores the subclasses of unittest.TestCase \r\n    _unitTestsList = []\r\n    \r\n    # Defines a regular expression that whitelist files that may contain unittest.TestCase\r\n    _testFilePathPattern = re.compile('.*[Tt]est.*\\.py$')\r\n    \r\n    def addLookupDirectoryPath(self, directoryPath):\r\n        self._directoryPathList.append(directoryPath)\r\n    \r\n    def gatherUnitTests(self):\r\n        \r\n        # For each indicated directory path\r\n        for directoryPath in self._directoryPathList:\r\n            \r\n            if os.path.isdir(directoryPath):\r\n                # Walk through the contents \r\n                for subFolderRoot, foldersWithinSubFolder, files in os.walk(directoryPath):\r\n                \r\n                   # look at the files for subclasses of unittest.TestCase\r\n                   for file in files:\r\n                       \r\n                       fileBasename = os.path.basename(file)\r\n                       if self._testFilePathPattern.match(fileBasename) :\r\n        \r\n                           # Import the relevant module (note: a module does not end with .py)\r\n                           moduleDirectory = os.path.join(subFolderRoot, os.path.splitext(file)[0])\r\n                           moduleStr = moduleDirectory.replace(os.path.sep, '.')\r\n                           module = importlib.import_module(moduleStr)\r\n        \r\n                           # Look for classes that implements unittest.TestCase\r\n                           for name in dir(module) :\r\n                               moduleAttribute = getattr(module, name)\r\n                               if isinstance(moduleAttribute, type) and issubclass(moduleAttribute, unittest.TestCase):\r\n                                   \r\n                                   # Use testLoader to load the tests that are defined in the unittest.TestCase class\r\n                                   self._unitTestsList.append(unittest.defaultTestLoader.loadTestsFromTestCase(moduleAttribute))\r\n\r\n            \r\n    def getUnitTestsList(self) : \r\n        return self._unitTestsList\r\n<\/pre>\n
      There are three functions in UnitTestFinder<\/code>:<\/p>\n
        \n
      1. The addLookupDirectoryPath<\/code> function.<\/li>\n
      2. The gatherUnitTests<\/code> function.<\/li>\n
      3. The getUnitTestsList<\/code> function.<\/li>\n<\/ol>\n
        What the addLookupDirectoryPath<\/code> function does<\/h5>\n
        The addLookupDirectoryPath<\/code> function remembers the specific sibling directories that we want our UnitTestFinder<\/code> to look into. The path to these directories are stored as list items via the _directoryPathList<\/code> variable.<\/p>\n
        What the gatherUnitTests<\/code> function does<\/h5>\n
        The gatherUnitTests<\/code> function then loop through the directory paths and begin searching for definition of subclasses of unittest.TestCase<\/code><\/a> inside each of the sibling directories. <\/p>\n
        Using the os.walk <\/code><\/a> function on each of the directory path that we added via the addLookupDirectoryPath<\/code> function, we check for files that start with the word 'test' or 'Test' and end with the '.py' extension via regular expression.<\/p>\n
        For Python files that match our regular expression, we use the importlib.import_module()<\/a> function to import the Python file and make it accessible via the module<\/code> variable.<\/p>\n
        We then loop through the attribute names in the module via the dir()<\/a> function. With the getattr()<\/a> function, we check for subclasses of the unittest.TestCase<\/code><\/a> class.<\/p>\n
        If there are subclasses of unittest.TestCase<\/code><\/a>, we use unittest.defaultTestLoader<\/code><\/a> to help us load the test cases within the subclasses. We then append whatever unittest.defaultTestLoader<\/code><\/a> returns us to self._unitTestsList<\/code>. <\/p>\n
        What the getUnitTestsList<\/code> function does<\/h5>\n
        The getUnitTestsList<\/code> functions returns the unit tests that the gatherUnitTests<\/code> function finds.<\/p>\n
        Building the code execution block that will run the unit tests that the Python class had gathered with an instance of unittest.TextTestRunner<\/code><\/h4>\n
        With UnitTestFinder<\/code> defined , I then proceed to writing the code block that will utilise UnitTestFinder<\/code> and run the test cases in one shot:<\/p>\n
        \r\nif __name__ == "__main__" :\r\n\r\n    unitTestFinder = UnitTestFinder()\r\n    unitTestFinder.addLookupDirectoryPath('folderWithTestCases')  \r\n    unitTestFinder.gatherUnitTests()\r\n    \r\n    # Execute the test cases that had been collected by UnitTestFinder\r\n    testSuite = unittest.TestSuite(unitTestFinder.getUnitTestsList())\r\n    testResult = unittest.TextTestRunner(verbosity=2).run(testSuite) \r\n\r\n    # Throw an error to notify error condition.\r\n    if not testResult.wasSuccessful() :\r\n        raise ValueError('There were test failures.') \r\n<\/pre>\n
        The code block starts with a test on whether the Python binary had ran our script first. If the script was ran first, we then:<\/p>\n
          \n
        1. Create an instance of UnitTestFinder<\/code>.<\/li>\n
        2. Specify the directory to look for test cases, which in our case is just the directory 'folderWithTestCases'.<\/li>\n
        3. Get the instance of UnitTestFinder<\/code> to gather the test cases that are located inside of 'folderWithTestCases'.<\/li>\n
        4. Create a unittest.TestSuite<\/code><\/a> instance from the test cases that UnitTestFinder<\/code> could gather and make it available via the testSuite<\/code> variable.<\/li>\n
        5. Get the test results by creating an instance of unittest.TextTestRunner<\/code><\/a> and using it to run our test suite. We capture the result with the testResult<\/code> variable.<\/li>\n
        6. Finally we check whether the test run was successful via a call to testResult.wasSuccessful()<\/code><\/a>. If not, we throw an instance of ValueError<\/code><\/a> to get Jenkins to send an email to all the developers in the team.<\/li>\n<\/ol>\n
          Putting everything together<\/h3>\n
          Putting the codes together, we will yield the following script:<\/p>\n
          \r\nimport importlib\r\nimport os\r\nimport re\r\nimport unittest\r\n\r\nclass UnitTestFinder:\r\n    \r\n    # Stores the list of directory paths that unittest.TestCase subclasses\r\n    # would reside in\r\n    _directoryPathList = []\r\n    \r\n    # Stores the subclasses of unittest.TestCase \r\n    _unitTestsList = []\r\n    \r\n    # Defines a regular expression that whitelist files that may contain unittest.TestCase\r\n    _testFilePathPattern = re.compile('.*[Tt]est.*\\.py$')\r\n    \r\n    def addLookupDirectoryPath(self, directoryPath):\r\n        self._directoryPathList.append(directoryPath)\r\n    \r\n    def gatherUnitTests(self):\r\n        \r\n        # For each indicated directory path\r\n        for directoryPath in self._directoryPathList:\r\n            \r\n            if os.path.isdir(directoryPath):\r\n                # Walk through the contents \r\n                for subFolderRoot, foldersWithinSubFolder, files in os.walk(directoryPath):\r\n                \r\n                   # look at the files for subclasses of unittest.TestCase\r\n                   for file in files:\r\n                       \r\n                       fileBasename = os.path.basename(file)\r\n                       if self._testFilePathPattern.match(fileBasename) :\r\n        \r\n                           # Import the relevant module (note: a module does not end with .py)\r\n                           moduleDirectory = os.path.join(subFolderRoot, os.path.splitext(file)[0])\r\n                           moduleStr = moduleDirectory.replace(os.path.sep, '.')\r\n                           module = importlib.import_module(moduleStr)\r\n        \r\n                           # Look for classes that implements unittest.TestCase\r\n                           for name in dir(module) :\r\n                               moduleAttribute = getattr(module, name)\r\n                               if isinstance(moduleAttribute, type) and issubclass(moduleAttribute, unittest.TestCase):\r\n                                   \r\n                                   # Use testLoader to load the tests that are defined in the unittest.TestCase class\r\n                                   self._unitTestsList.append(unittest.defaultTestLoader.loadTestsFromTestCase(moduleAttribute))\r\n\r\n            \r\n    def getUnitTestsList(self) : \r\n        return self._unitTestsList\r\n\r\nif __name__ == "__main__" :\r\n\r\n    unitTestFinder = UnitTestFinder()\r\n    unitTestFinder.addLookupDirectoryPath('folderWithTestCases')  \r\n    unitTestFinder.gatherUnitTests()\r\n    \r\n    # Execute the test cases that had been collected by UnitTestFinder\r\n    testSuite = unittest.TestSuite(unitTestFinder.getUnitTestsList())\r\n    testResult = unittest.TextTestRunner(verbosity=2).run(testSuite) \r\n\r\n    # Throw an error to notify error condition.\r\n    if not testResult.wasSuccessful() :\r\n        raise ValueError('There were test failures.')\r\n<\/pre>\n
          Trying out the script<\/h5>\n
          To try out the script, I created a sample Python script, 'test_arithmetic_operations_a.py', and place it inside the directory, 'folderWithTestCases':<\/p>\n
          \r\nimport unittest\r\n\r\nclass ArithmeticTestCases(unittest.TestCase) :\r\n\r\n    def test_addition(self):\r\n        self.assertEqual(1+1, 2)\r\n        \r\n    def test_multiplication(self):\r\n        self.assertEqual(2*3, 6)\r\n<\/pre>\n
          And another Python script, 'test_arithmetic_operations_b.py' and place it inside the directory, 'folderWithTestCases\/anotherFolder':<\/p>\n
          \r\nimport unittest\r\n\r\nclass ArithmeticTestCasesB(unittest.TestCase) :\r\n\r\n    def test_addition(self):\r\n        self.assertEqual(1+1, 2)\r\n        \r\n    def test_multiplication(self):\r\n        self.assertEqual(2*3, 6)\r\n<\/pre>\n
          I then run the following command inside of the directory that contains the Python script that I had written earlier:<\/p>\n
          \r\npython3 run_unit_tests_in_one_shot.py \r\n<\/pre>\n
          Which produced the following output:<\/p>\n
          \r\ntest_addition (folderWithTestCases.test_arithmetic_operations_a.ArithmeticTestCasesA) ... ok\r\ntest_multiplication (folderWithTestCases.test_arithmetic_operations_a.ArithmeticTestCasesA) ... ok\r\ntest_division (folderWithTestCases.anotherFolder.test_arithmetic_operations_b.ArithmeticTestCasesB) ... ok\r\ntest_subtraction (folderWithTestCases.anotherFolder.test_arithmetic_operations_b.ArithmeticTestCasesB) ... ok\r\n\r\n----------------------------------------------------------------------\r\nRan 4 tests in 0.001s\r\n\r\nOK\r\n<\/pre>\n
          I then change the test_addition test case by asserting that 1+1 equals 3 in order to create a test failure:<\/p>\n
          \r\nimport unittest\r\n\r\nclass ArithmeticTestCasesA(unittest.TestCase) :\r\n\r\n    def test_addition(self):\r\n        self.assertEqual(1+1, 3)\r\n        \r\n    def test_multiplication(self):\r\n        self.assertEqual(2*3, 6)\r\n<\/pre>\n
          This time, I got the following output:<\/p>\n
          \r\ntest_addition (folderWithTestCases.test_arithmetic_operations_a.ArithmeticTestCasesA) ... FAIL\r\ntest_multiplication (folderWithTestCases.test_arithmetic_operations_a.ArithmeticTestCasesA) ... ok\r\ntest_division (folderWithTestCases.anotherFolder.test_arithmetic_operations_b.ArithmeticTestCasesB) ... ok\r\ntest_subtraction (folderWithTestCases.anotherFolder.test_arithmetic_operations_b.ArithmeticTestCasesB) ... ok\r\n\r\n======================================================================\r\nFAIL: test_addition (folderWithTestCases.test_arithmetic_operations_a.ArithmeticTestCasesA)\r\n----------------------------------------------------------------------\r\nTraceback (most recent call last):\r\n  File "\/techcoil\/poc\/FindAndRunUnitTestsAtOneGoProject\/folderWithTestCases\/test_arithmetic_operations_a.py", line 6, in test_addition\r\n    self.assertEqual(1+1, 3)\r\nAssertionError: 2 != 3\r\n\r\n----------------------------------------------------------------------\r\nRan 4 tests in 0.001s\r\n\r\nFAILED (failures=1)\r\nTraceback (most recent call last):\r\n  File "run_unit_tests_in_one_shot.py", line 65, in <module>\r\n    raise ValueError('There were test failures.')    \r\nValueError: There were test failures.\r\n<\/pre>\n\n