Exception handling and debugging in Python

You have probably already seen Python’s exception handling in action from trying to execute invalid syntax, or some other illegal operation. For instance:

>> 1/0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero

The first part in the last line says ZeroDivisionError. This is one of the Python’s built-in exceptions for arithmetic errors.

Exceptions are raised by different kinds of errors when executing invalid Python code. You may also throw your own errors, and even define custom error types specific to your code base. Please see a more complete list of Python’s built-in exceptions at https://docs.python.org/3/library/exceptions.html.

Below, we shall take a look at some examples which will teach us to how to use exception handling in Python programs. In addition, we are going to study a few debugging strategies to help deal with exceptions that arise when you aren’t expecting them.

Exceptions

A few of the more common built in exceptions are listed here:

• TypeError: unsupported operation

  >>> 1+'apple'
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  TypeError: unsupported operand type(s) for +: 'int' and 'str'
  

• KeyError: invalid use of key

>>> eng2kor = {'three': 'set', 'two': 'dool', 'one': 'hana'}
>>> eng2kor[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 0

• IndexError: invalid use of index

  >>> a = [1, 2, 3]
  >>> a[4]
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  IndexError: list index out of range
  

• AttributeError: attribute reference or assignment failure

  >>> eng2kor.append('foo')
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  AttributeError: 'dict' object has no attribute 'append'
  

• SyntaxError: the most basic type of error when the Python parser is unable to understand a line of code

  >>> print "I love scientific computing"
  SyntaxError: Missing parentheses in call to 'print'. Did you mean print("I love scientific computing")?
  

  The above syntax error due to missing parentheses happens mostly when migrating Python2 codes to Python3.

Catching exceptions

try/except

The try statement works as follows:

1. Python attempts to execute the statement(s) between try and except

2. If no exception occurs, the try block terminates and the except block is ignored

3. If an exception does occur during the execution of the try block, the rest of the block is skipped, and the exception is passed to the except block.

4. If its type matches the exception named after the except keyword, the body of the except clause is executed, and then execution continues after the try-except statement.

5. If the type does not match, then the exception will get forwarded to additional except blocks if they are present. Otherwise, it gets passed up a layer (potentially all the way to the interpreter). If the interpreter catches the exception then you’ll see one of the above messages: Exceptions.

Now let’s take a look at few examples. In the first example, you will see the error message if you enter an input that is not a number:

"""

lectureNote/chapters/chapt05/codes/try_except1.py

try-except example, originally from https://docs.python.org/3/tutorial/errors.html

"""


while True:
    try:
        x = int(input("Please enter an integer: "))
        print('The number you entered was = ', x)
    except ValueError:
        print("Oops! Could not invert your input to an integer. Try again...")

Download this code

Exception handlers don’t just handle exceptions when they occur immediately within the try clause, but also when they occur inside functions that are called (even indirectly) in the try clause. For example:

"""

lectureNote/chapters/chapt05/codes/try_except2.py

try-except example, originally from https://docs.python.org/3/tutorial/errors.html

"""

def this_fails():
    print('try-except example')
    print('1. valid division 1/2')
    print('2. invalid division 1/0')
    option = int(input("Please enter an option number -- 1 or 2: "))

    if option == 1:
        x = 1./2
    elif option == 2:
        x = 1./0
    return x

try:
    x = this_fails()
    print(x)
except ZeroDivisionError:
    print('Handling run-time error')

Download this code

try/finally

A finally block can optionally be used with a try/except pair, which is intended to define clean-up actions that must be executed under all circumstances. For instance:

"""

lectureNote/chapters/chapt05/codes/try_finally.py

try-finally example, originally from https://docs.python.org/3/tutorial/errors.html

"""

def divide(x, y):
     try:
         result = x / y
         print("Result is", result)
     except ZeroDivisionError:
         print("Division by zero!")
     finally:
         print("Executing finally clause, Thanks for testing me!")

if __name__ == '__main__':
     print("calling as divide(1.0,2.0)")
     divide(1.0,2.0)
     print("calling as divide(1.0,0.0)")
     divide(1.0,0.0)

Download this code

Such a clean-up step can be useful for resource management, e.g., closing a file.

Raising exceptions

Let’s now take a look at how to raise an exception in your code. In the following example, we run an iteration using our old example of Newton’s root finding method. One idea is to raise an exception of ArithmeticError (see line 33 in the example below) when the method fails to converge.

"""

lectureNote/chapters/chapt04/codes/try_except3.py

try-except example, written by Prof. Dongwook Lee for AM 129/209.

Modified by Ian May, FA2021

"""

import numpy as np

def fdf(x):
    f  = x  + np.exp(x) + 10/(1 + x**2) - 5
    df = 1.0 + np.exp(x) - 20*x/(1 + x**2)**2
    return f,df

def root_finder(func, x0, thresh, MaxIter=100):
    # Put initial guess in list to store history
    x = [x0]
    # Query function at initial guess
    f,df = func(x0)
    res = [abs(f)]
    for i in range(0,MaxIter):
        # Raise error if slope is very flat
        if (abs(df)<1.e-5):
            raise ArithmeticError
        # Newton's iteration
        x.append(x[-1] - f/df)
        # Query function at new guess
        f,df = func(x[-1])
        # Update residual
        res.append(abs(f))
        print(x[-1],f,df)
        # Check for convergence
        if (res[-1] < thresh):
            break
    
    # If final residual is too big we stagnated
    if (res[-1] > thresh):
        raise ArithmeticError

    # Return sequence of guesses and residuals
    return x,res


if __name__ == '__main__':
    # take an initial value as an input from users
    x0 = float(input("Please enter an initial search value: "))
    # take a threshold value as an input from users
    threshold = float(input("Please enter a threshold value: "))
    # Call Newton's method
    try:
        x, res = root_finder(fdf,x0,threshold)
    except ArithmeticError:
        print("Newton's method failed for that initial guess")
    else:
        print("Newton's method converged in %d iterations" % len(x))
        for v,r in zip(x,res):
            print(v, r)

Download this code

Try running this code using the initial guesses of \(x=-1/2,0,1,1.1634\).

Python debugging

Print statements

Print statements can be added almost anywhere in a Python code to print things out to the terminal window as it goes along.

You might want to put some special symbols in debugging statements to flag them as such, which makes it easier to see what output is your debug output and also makes it easier to find them again later to remove from the code, e.g. you might use “+++” or “DEBUG”.

As an example, suppose you are trying to better understand Python scoping and namespaces, as well as the difference between local and global variables. Then this code might be useful:

"""
lectureNote/chapters/chapt05/codes/debugging1.py

Originally from
http://faculty.washington.edu/rjl/classes/am583s2014/notes/python_debugging.html

Debugging demo using print statements

"""

x = 3.5
y = -22.231

def f(z):
    global y
    x = z + 10
    y = 20.19
    print("+++ in function f: x = %s, y = %s, z = %s, tt=%s" % (x,y,z,'i love am129/209')) # %s also works for numbers
    # print("+++ in function f: x = %f, y = %f, z = %f, tt=%s" % (x,y,z,'i love am129/209')) # but in general, %f also works for all numbers
    return x

print("+++ before calling f: x = %s, y = %s" % (x,y))
y1 = f(x)
print("+++ after calling f: x = %s, y = %s, y1= %s" % (x,y,y1))

Download this code

Note

In the above example, we see two tokens, %s for strings and %d for numbers, as placeholders of what comes after the % sign, e.g., % (x, y, z). See more examples here. We’ll see more about this when we cover C.

Here the print function in the definition of f(x) is being used for debugging purposes. Executing this code gives

$ python3 debugging1.py
+++ before calling f: x = 3.0, y = -22.0
+++ in function f: x = 13.0, y = -22.0, z = 3.0
+++ after calling f: x = 3.0, y = 13.0

If you are printing large amounts you might want to write to a file rather than to the terminal.

The pdb debugger

Inserting print statements may work best in some situations (particularly for debugging logic errors), but it is often better to use a debugger. The Python debugger pdb is very easy to use, often even easier than inserting print statements and is well worth learning. See the pdb documentation for more information. The syntax is very similar to gdb which we discussed before.

You can insert breakpoints in your code (see line 17 in the below) where control should pass back to the user, at which point you can query the value of any variable, or step through the program line by line from this point on:

"""
lectureNote/chapters/chapt05/codes/debugging2.py

Originally from
http://faculty.washington.edu/rjl/classes/am583s2014/notes/python_debugging.html

Debugging demo using print statements

Debugging demo using pdb.
"""

import pdb

x = 3.
y = -22.

def f(z):
    x = z+10
    pdb.set_trace()
    return x

y = f(x)

print("x = ",x)
print("y = ",y)

Download this code

Of course one could set multiple breakpoints with other pdb.set_trace() commands. For the above example we might do this as below. Upon running the above example, we get the prompt for the pdb shell when we hit the breakpoint

$ python3 debugging2.py

> /home/ian/UCSC/AM129Lectures/Python/debugging2.py(18)f()
-> return x
(Pdb) p x
13.0
(Pdb) p y
-22.0
(Pdb) p z
3.0

Note that p is short for print which is the same as gdb command. You could also type print(x) but this would then execute the Python print command instead of the debugger command (though in this case it would print the same thing).

There are many other pdb commands, such as next to execute the next line, continue to continue executing until the next breakpoint, etc.

$ python3 debugging2.py

> /home/ian/UCSC/AM129Lectures/Python/debugging2.py(18)f()
-> return x
(Pdb) p z
3.0
(Pdb) continue
x =  3.0
y =  13.0

(See the pdb documentation for more details.)

Debugging after an exception occurs

Often code has bugs that cause an exception to be raised, causing the program to halt execution. Consider the following example:

"""

lectureNote/chapters/chapt05/codes/pdb_example.py

"""
        
def division_by_zero(x):
    x /= x-1  # equiv. to x = x/(x-1)
    return x

# import pdb
for i in range(5,0,-1):
    # pdb.set_trace()
    soln = division_by_zero(float(i))

Download this code

In IPython you can run this by calling

>>> %run pdb_example.py
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
~/UCSC/AM129/LectureF21/Python/Fundamentals/pdb_example.py in <module>
     12 for i in range(5,0,-1):
     13     # pdb.set_trace()
---> 14     soln = division_by_zero(float(i))

~/UCSC/AM129/LectureF21/Python/Fundamentals/pdb_example.py in division_by_zero(x)
      6
      7 def division_by_zero(x):
----> 8     x/=x-1  # equiv. to x=x/x-1
      9     return x
     10

ZeroDivisionError: float division by zero

At some point there must be a division by zero. To figure out when this happens, we could insert a pdb.set_trace() command in the loop and step through it until the error occurs and then look at i. Alternatively, and more easily, we can use a post-mortem analysis after it dies via pdb.pm()

>>> import pdb
>>> pdb.pm()
> <string>(8)division_by_zero()
(Pdb) p i
1
(Pdb) p soln
2.0
(Pdb) p x
1.0

This starts up pdb at exactly the point where the exception is about to occur. We see that the divide by zero happens when i = 1. Compare this to our usage of bt in gdb.