Fault Injection

Problem

You don't have control over external libraries but you need to test how your software handles errors from them.

Solution

Create duck-typed substitutes that call the external libraries but sometimes introduce errors. The percentage of errors should be low so that it doesn't interfere with other operations.
From Udacity's Software Testing course.

Specifications and Testing

Goal

To treat the software under test as a black box by testing to the Application Programming Interface (API).

What role does testing play in the specification of software?

Part of the job of testing is to help refine the specification by figuring out what acceptable inputs and outputs are.

What are Domains and Ranges?

Domain: the set of possible inputs Range: the set of possible outputs

What are Software Domains and Ranges?

Software uses a super-set of the mathematical domains and ranges.

  • Domain: All inputs of the right type (or all inputs) 
  • Range: Union of valid range with a set of exceptions

Should You Use Defensive Coding?

  • Defensive Coding means to validate the input that users give you before proceeding.
  • Implementing checks for valid input makes the code slow and hard to read
  • Better to clearly specify the Domain in the documentation and raise an exception if an invalid output is produced.
From Udacity's Software Testing Unit 1.

More On Assertions

Why Use Assertions?

  1. The code becomes self-checking.
  2. The code fails at a location closer to the bug.
  3. If you put assertions near the interfaces between modules, it makes it easier to assign blame.
  4. It creates executable documentation about:
    • preconditions
    • invariants
    • postconditions

Assertions In Production Code

Production Compilers
Compiler Assertion Count
GCC ~9000
LLVM ~13000
Total lines of code for LLVM: ~1,400,000. Ratio for LLVM of Assertions to Code:
1/110

Disabling Assertions

Advantages
  • Runs faster
  • Doesn't abort execution on error
Disadvantages
  • If the Assertion accidentally had side-effects, disabling it will change the behavior
  • Often better to fail early rather than operate incorrectly
Conclusion
  • It depends on what you want:
    • fail early and alert user to problem
    • keep going and accept risk of incorrect output

When To Use Assertions

  • If you're doing something as critical as the final stages of a rocket landing where aborting could destroy something, it's okay to turn them off.
  • Otherwise don't
From Udacity's Software Testing Unit 1.

A Tester for the Check Rep

from unittest import TestCase
import sys
from random import randint

from nose.tools import raises

from softwaretesting import fixedsizequeue


MAXINT = sys.maxint
MININT = -sys.maxint - 1


class TestFixedCheckRep(TestCase):
def setUp(self):
self.size = randint(1, 100)
self.items = [randint(MININT, MAXINT) for item in range(self.size)]
self.q = fixedsizequeue.FixedsizeQueue(self.size)
return

def check_fail_reset(self):
"""
Checks if the check_rep fails then resets the q
"""

print("Testing: {0}".format(self.q))
self.assertRaises(AssertionError, self.q.check_representation)
self.q.reset()
print("Passed: {0}".format(self.q))
return

def test_okay(self):
"""
The enqueue takes a single integer and returns True if added.
"""

self.q.check_representation()
return

@raises(AssertionError)
def test_enqueue_wraparound_error(self):
"""
The tail should wrap around when it fills
"""

self.q.reset()
for item in self.items:
self.q.enqueue(item)
self.q.tail = 1
self.q.check_representation()
return

@raises(AssertionError)
def test_dequeue_wraparound_error(self):
"""
After emptying the queue, the head should be 0
"""

self.q.reset()
for item in self.items:
self.q.enqueue(item)
for item in self.items:
self.q.dequeue()
self.q.head = 1
self.q.check_representation()
return

def test_negative_error(self):
"""
The properties of the queue should never become negative.
"""

self.q.reset()
value = -1 * abs(self.items[0])
self.q.check_representation()
self.q.head = value
self.check_fail_reset()
self.q.tail = value
self.check_fail_reset()
self.q.head = value
self.check_fail_reset()


self.q.size = value
self.check_fail_reset()

self.q.max = value
self.check_fail_reset()
return

def test_negative_offset_error(self):
self.q.reset()
self.q.check_representation()
value = -1 * abs(self.items[0])
self.q.head += value
self.q.size -= value
self.check_fail_reset()

self.q.head -= value
self.q.size += value
self.check_fail_reset()
return
# end class TestCheckRep

A Python Implementation Of the CheckRep

The method was implementated as:
def check_representation(self):
"""
Checks that the internal representation is correct.

:raise: AssertionError if an error is found.
"""
for attribute in (self.max, self.head, self.size):
assert attribute >= 0, "{0} shouldn't be negative. ({1})".format(attribute, str(self))
assert ((self.head + self.size) % self.max == self.tail), str(self)
return

Notes

  • This is for the fixed-size queue used in udacity's Software Testing class
  • Since the second assertion checks an equation where the tail can never be negative if all the left-hand terms are positive, the tail isn't checked to see if it's positive

CheckRep

What is checkRep?

  • The name is short for check representation
  • It's a method that can be called to check if the current state of the data-structure is valid.

What does it do?

  • It calls a series of assertions that validate the properties of the object to which it belongs.
  • The assertions test invariants.

What are 'invariants'?

  • Invariants are conditions that must always be true for the algorithm to be correct.
From Udacity's Software Testing course -- Unit 1.

The Bag ADT [DSAUP]

What is a Bag?

A bag:
  • is a collection of things
  • allows duplicates
  • isn't ordered but the items should be comparable

The Interface

The Bag
Method Effect
Bag() Creates an empty bag
length() Returns the number of items in the bag
contains(item) Returns True if item is in the bag
add(item) Adds item to the bag
remove(item) Remove and return item from the bag or raise exception if it's not in the bag
iterator() Create and return iterator to iterate over the bag

Assertions For Testable Code

What is an assertion?

An assertion is an executable check for a property that must be true of the code.

Rule 1

Assertions Aren't For Error Handling (that's what exceptions are for)
  • Assert something about the result of your logic
  • Don't assert anything about the result of another entity's logic

Rule 2

No Side Effects
  • Assertions shouldn't change the state of any variables outside the scope of the assertion

Rule 3

No Silly Assertions
  • Check for errors in your own logic
  • Don't check if the user did something wrong (in the assertion)
  • Don't check that any other entity is working correctly
From Udacity's Software Testing course notes.

Creating Testable Software

  1. Create Clean Code -- code for which you can:
    • describe exactly what it does
    • describe exactly how it interacts with other entities
  2. Minimize Threads
  3. Minimize Global Variables -- these create implicit inputs to any software in the same module.
  4. Reduce references and pointers.
  5. Maximize Assertions
From Udacity's Testing Software course.

Testing the Fixed-Size Queue

from unittest import TestCase
import sys
from random import randint

from nose.tools import raises

from softwaretesting import fixedsizequeue


MAXINT = sys.maxint
MININT = -sys.maxint - 1


class TestFixedSizeQueue(TestCase):
def setUp(self):
self.size = 10
self.items = [randint(MININT, MAXINT) for item in range(self.size)]
self.q = fixedsizequeue.FixedsizeQueue(self.size)
return

def test_enqueue(self):
"""
The enqueue takes a single integer and returns True if added.
"""

self.q.reset()
self.assertFalse(self.q.full())
for index, item in enumerate(self.items):
self.assertTrue(self.q.enqueue(item))
self.assertEqual(index + 1, self.q.size)
self.assertEqual(item, self.q.data[index])
self.assertEqual((index + 1) % len(self.items), self.q.tail)
self.assertFalse(self.q.empty())
self.assertTrue(self.q.full())
return

def test_dequeue(self):
"""
The dequeue returns the oldest item or None.
"""

self.q.reset()
self.assertIsNone(self.q.dequeue())
self.assertEqual(0, self.q.size)
for item in self.items:
self.q.enqueue(item)

for expected in self.items:
actual = self.q.dequeue()
self.assertEqual(expected, actual)
self.assertTrue(self.q.empty())
self.assertEqual(0, self.q.head)
return

def test_reset(self):
"""
The reset resets the pointers and count
"""

self.q.enqueue(9)
self.q.enqueue(8)
self.q.dequeue()
self.q.reset()
self.assertTrue(self.q.empty())
self.assertEqual(0, self.q.size)
self.assertEqual(0, self.q.tail)
self.assertEqual(0, self.q.head)
return

def test_empty(self):
"""
If the queue is empty dequeue returns False and does nothing.
"""

q = fixedsizequeue.FixedsizeQueue(3)
self.assertTrue(q.empty())
q.enqueue(randint(MININT, MAXINT))
self.assertFalse(q.empty())
q.dequeue()
self.assertTrue(q.empty())
return

def test_overfull(self):
"""
The array doesn't add items once full
"""

self.q.reset()

for item in self.items:
self.q.enqueue(item)
self.assertEqual(0, self.q.tail)
self.assertTrue(self.q.full())
self.assertEqual(len(self.items), self.q.size)
self.assertFalse(self.q.enqueue(1))
self.assertEqual(len(self.items), self.q.size)
self.assertEqual(self.q.size, self.q.max)

for item in self.items:
self.assertEqual(item, self.q.dequeue())
self.assertTrue(self.q.empty())

return

@raises(TypeError)
def test_non_int(self):
"""
The array has to be homogeneous.
"""

self.q.enqueue(1.0)
return
# end class TestFixedSizeQueue