Testing

• assert statement
• Using assert to test a program
• Using unittest framework
• Using unittest.mock to test user input and program output
• Other testing frameworks

assert statement

• assert is primarily used for debugging purposes like catching invalid input or a condition that shouldn't occur
• An optional message can be passed for descriptive error message than a plain AssertionError
• It uses raise statement for implementation
• assert statements can be skipped by passing the -O command line option
• Note that assert is a statement and not a function

>>> assert 2 ** 3 == 8
>>> assert 3 > 4
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError

>>> assert 3 > 4, "3 is not greater than 4"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError: 3 is not greater than 4

Let's take a factorial function as an example:

>>> def fact(n):
        total = 1
        for num in range(1, n+1):
            total *= num
        return total

>>> assert fact(4) == 24
>>> assert fact(0) == 1
>>> fact(5)
120

>>> fact(-3)
1
>>> fact(2.3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in fact
TypeError: 'float' object cannot be interpreted as an integer

• assert fact(4) == 24 and assert fact(0) == 1 can be considered as sample tests to check the function

Let's see how assert can be used to validate arguments passed to the function:

>>> def fact(n):
        assert type(n) == int and n >= 0, "Number should be zero or positive integer"
        total = 1
        for num in range(1, n+1):
            total *= num
        return total

>>> fact(5)
120
>>> fact(-3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in fact
AssertionError: Number should be zero or positive integer
>>> fact(2.3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in fact
AssertionError: Number should be zero or positive integer

The above factorial function can also be written using reduce

>>> def fact(n):
        assert type(n) == int and n >= 0, "Number should be zero or positive integer"
        from functools import reduce
        from operator import mul
        return reduce(mul, range(1, n+1), 1)


>>> fact(23)
25852016738884976640000

Above examples for demonstration only, for practical purposes use math.factorial which also gives appropriate exceptions

>>> from math import factorial
>>> factorial(10)
3628800

>>> factorial(-5)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: factorial() not defined for negative values
>>> factorial(3.14)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: factorial() only accepts integral values

Further Reading

• Python docs - assert
• What is the use of assert in Python?
• Is Unit Testing worth the effort?

Using assert to test a program

In a limited fashion, one can use assert to test a program - either within the program (and later skipped using the -O option) or as separate test program(s)

Let's try testing the palindrome program we saw in Docstrings chapter

#!/usr/bin/python3

import palindrome

assert palindrome.is_palindrome('Madam')
assert palindrome.is_palindrome("Dammit, I'm mad!")
assert not palindrome.is_palindrome('aaa')
assert palindrome.is_palindrome('Malayalam')

try:
    assert palindrome.is_palindrome('as2')
except ValueError as e:
    assert str(e) == 'Characters other than alphabets and punctuations'

try:
    assert palindrome.is_palindrome("a'a")
except ValueError as e:
    assert str(e) == 'Less than 3 alphabets'

print('All tests passed')

• There are four different cases tested for is_palindrome function
  • Valid palindrome string
  • Invalid palindrome string
  • Invalid characters in string
  • Insufficient characters in string
• Both the program being tested and program to test are in same directory
• To test the main function, we need to simulate user input. For this and other useful features, test frameworks come in handy

$ ./test_palindrome.py 
All tests passed

Using unittest framework

This section requires understanding of classes

#!/usr/bin/python3

import palindrome
import unittest

class TestPalindrome(unittest.TestCase):

    def test_valid(self):
        # check valid input strings
        self.assertTrue(palindrome.is_palindrome('kek'))
        self.assertTrue(palindrome.is_palindrome("Dammit, I'm mad!"))
        self.assertFalse(palindrome.is_palindrome('zzz'))
        self.assertFalse(palindrome.is_palindrome('cool'))

    def test_error(self):
        # check only the exception raised
        with self.assertRaises(ValueError):
            palindrome.is_palindrome('abc123')

        with self.assertRaises(TypeError):
            palindrome.is_palindrome(7)

        # check error message as well
        with self.assertRaises(ValueError) as cm:
            palindrome.is_palindrome('on 2 no')
        em = str(cm.exception)
        self.assertEqual(em, 'Characters other than alphabets and punctuations')

        with self.assertRaises(ValueError) as cm:
            palindrome.is_palindrome('to')
        em = str(cm.exception)
        self.assertEqual(em, 'Less than 3 alphabets')

if __name__ == '__main__':
    unittest.main()

• First we create a subclass of unittest.TestCase (inheritance)
• Then, different type of checks can be grouped in separate functions - function names starting with test are automatically called by unittest.main()
• Depending upon type of test performed, assertTrue, assertFalse, assertRaises, assertEqual, etc can be used
• An Introduction to Classes and Inheritance
• Python docs - unittest
  • Command-Line Interface
  • Test Discovery
  • Organizing test code, setUp and tearDown

$ ./unittest_palindrome.py
..
----------------------------------------------------------------------
Ran 2 tests in 0.001s

OK

$ ./unittest_palindrome.py -v
test_error (__main__.TestPalindrome) ... ok
test_valid (__main__.TestPalindrome) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.001s

OK

Using unittest.mock to test user input and program output

This section requires understanding of decorators, do check out this wonderful intro

A simple example to see how to capture print output for testing

>>> from unittest import mock
>>> from io import StringIO

>>> def greeting():
        print('Hi there!')

>>> def test():
        with mock.patch('sys.stdout', new_callable=StringIO) as mock_stdout:
            greeting()
            assert mock_stdout.getvalue() == 'Hi there!\n'

>>> test()

One can also use decorators

>>> @mock.patch('sys.stdout', new_callable=StringIO)
    def test(mock_stdout):
        greeting()
        assert mock_stdout.getvalue() == 'Hi there!\n'

Now let's see how to emulate input

>>> def greeting():
        name = input('Enter your name: ')
        print('Hello', name)

>>> greeting()
Enter your name: learnbyexample
Hello learnbyexample

>>> with mock.patch('builtins.input', return_value='Tom'):
        greeting()

Hello Tom

Combining both

>>> @mock.patch('sys.stdout', new_callable=StringIO)
    def test_greeting(name, mock_stdout):
        with mock.patch('builtins.input', return_value=name):
            greeting()
            assert mock_stdout.getvalue() == 'Hello ' + name + '\n'

>>> test_greeting('Jo')

Having seen basic input/output testing, let's apply it to main function of palindrome

#!/usr/bin/python3

import palindrome
import unittest
from unittest import mock
from io import StringIO

class TestPalindrome(unittest.TestCase):

    @mock.patch('sys.stdout', new_callable=StringIO)
    def main_op(self, tst_str, mock_stdout):
        with mock.patch('builtins.input', side_effect=tst_str):
            palindrome.main()
        return mock_stdout.getvalue()

    def test_valid(self):
        for s in ('Malayalam', 'kek'):
            self.assertEqual(self.main_op([s]), s + ' is a palindrome\n')

        for s in ('zzz', 'cool'):
            self.assertEqual(self.main_op([s]), s + ' is NOT a palindrome\n')

    def test_error(self):
        em1 = 'Error: Characters other than alphabets and punctuations\n'
        em2 = 'Error: Less than 3 alphabets\n'

        tst1 = em1 + 'Madam is a palindrome\n'
        self.assertEqual(self.main_op(['123', 'Madam']), tst1)

        tst2 = em2 + em1 + 'Jerry is NOT a palindrome\n'
        self.assertEqual(self.main_op(['to', 'a2a', 'Jerry']), tst2)

if __name__ == '__main__':
    unittest.main()

• Two test functions - one for testing valid input strings and another to check error messages
• Here, side_effect which accepts iterable like list, compared to return_value where only one input value can be mocked
• For valid input strings, the palindrome main function would need only one input value
• For error conditions, the iterable comes handy as the main function is programmed to run infinitely until valid input is given
• Python docs - unittest.mock
  • patchers
• An Introduction to Mocking in Python
• PEP 0318 - decorators
• decorators

$ ./unittest_palindrome_main.py 
..
----------------------------------------------------------------------
Ran 2 tests in 0.003s

OK

Other testing frameworks

• pytest
• Python docs - doctest
• Python test automation
• Python Testing Tools Taxonomy
• Python test frameworks

Test driven development (TDD)

• Test-Driven Development with Python
• learn Python via TDD