# Fault Injection

#### Problem

You don't have control over external libraries but you need to test how your software handles errors from them.

#### Solution

Create duck-typed substitutes that call the external libraries but sometimes introduce errors. The percentage of errors should be low so that it doesn't interfere with other operations.
From Udacity's Software Testing course.

# Specifications and Testing

#### Goal

To treat the software under test as a black box by testing to the Application Programming Interface (API).

#### What role does testing play in the specification of software?

Part of the job of testing is to help refine the specification by figuring out what acceptable inputs and outputs are.

#### What are Domains and Ranges?

Domain: the set of possible inputs Range: the set of possible outputs

#### What are Software Domains and Ranges?

Software uses a super-set of the mathematical domains and ranges.

• Domain: All inputs of the right type (or all inputs)
• Range: Union of valid range with a set of exceptions

#### Should You Use Defensive Coding?

• Defensive Coding means to validate the input that users give you before proceeding.
• Implementing checks for valid input makes the code slow and hard to read
• Better to clearly specify the Domain in the documentation and raise an exception if an invalid output is produced.
From Udacity's Software Testing Unit 1.

# More On Assertions

#### Why Use Assertions?

1. The code becomes self-checking.
2. The code fails at a location closer to the bug.
3. If you put assertions near the interfaces between modules, it makes it easier to assign blame.
4. It creates executable documentation about:
• preconditions
• invariants
• postconditions

#### Assertions In Production Code

Production Compilers
Compiler Assertion Count
GCC ~9000
LLVM ~13000
Total lines of code for LLVM: ~1,400,000. Ratio for LLVM of Assertions to Code:
1/110

#### Disabling Assertions

• Runs faster
• Doesn't abort execution on error
• If the Assertion accidentally had side-effects, disabling it will change the behavior
• Often better to fail early rather than operate incorrectly
##### Conclusion
• It depends on what you want:
• fail early and alert user to problem
• keep going and accept risk of incorrect output

#### When To Use Assertions

• If you're doing something as critical as the final stages of a rocket landing where aborting could destroy something, it's okay to turn them off.
• Otherwise don't
From Udacity's Software Testing Unit 1.

# A Tester for the Check Rep

from unittest import TestCaseimport sysfrom random import randintfrom nose.tools import raisesfrom softwaretesting import fixedsizequeueMAXINT = sys.maxintMININT = -sys.maxint - 1class TestFixedCheckRep(TestCase):def setUp(self):self.size = randint(1, 100)self.items = [randint(MININT, MAXINT) for item in range(self.size)]self.q = fixedsizequeue.FixedsizeQueue(self.size)returndef check_fail_reset(self):"""        Checks if the check_rep fails then resets the q        """print("Testing: {0}".format(self.q))self.assertRaises(AssertionError, self.q.check_representation)self.q.reset()print("Passed: {0}".format(self.q))returndef test_okay(self):"""        The enqueue takes a single integer and returns True if added.        """self.q.check_representation()return@raises(AssertionError)def test_enqueue_wraparound_error(self):"""        The tail should wrap around when it fills        """self.q.reset()for item in self.items:self.q.enqueue(item)self.q.tail = 1self.q.check_representation()return@raises(AssertionError)def test_dequeue_wraparound_error(self):"""        After emptying the queue, the head should be 0        """self.q.reset()for item in self.items:self.q.enqueue(item)for item in self.items:self.q.dequeue()self.q.head = 1self.q.check_representation()returndef test_negative_error(self):"""        The properties of the queue should never become negative.        """self.q.reset()value = -1 * abs(self.items[0])self.q.check_representation()self.q.head = valueself.check_fail_reset()self.q.tail = valueself.check_fail_reset()self.q.head = valueself.check_fail_reset()self.q.size = valueself.check_fail_reset()self.q.max = valueself.check_fail_reset()returndef test_negative_offset_error(self):self.q.reset()self.q.check_representation()value  = -1 * abs(self.items[0])self.q.head += valueself.q.size -= valueself.check_fail_reset()self.q.head -= valueself.q.size += valueself.check_fail_reset()return# end class TestCheckRep

# A Python Implementation Of the CheckRep

The method was implementated as:
def check_representation(self):    """    Checks that the internal representation is correct.    :raise: AssertionError if an error is found.    """    for attribute in (self.max, self.head, self.size):        assert attribute >= 0, "{0} shouldn't be negative. ({1})".format(attribute, str(self))    assert ((self.head + self.size) % self.max == self.tail), str(self)    return

#### Notes

• This is for the fixed-size queue used in udacity's Software Testing class
• Since the second assertion checks an equation where the tail can never be negative if all the left-hand terms are positive, the tail isn't checked to see if it's positive

# CheckRep

#### What is checkRep?

• The name is short for check representation
• It's a method that can be called to check if the current state of the data-structure is valid.

#### What does it do?

• It calls a series of assertions that validate the properties of the object to which it belongs.
• The assertions test invariants.

#### What are 'invariants'?

• Invariants are conditions that must always be true for the algorithm to be correct.
From Udacity's Software Testing course -- Unit 1.

#### What is a Bag?

A bag:
• is a collection of things
• allows duplicates
• isn't ordered but the items should be comparable

#### The Interface

The Bag
Method Effect
Bag() Creates an empty bag
length() Returns the number of items in the bag
contains(item) Returns True if item is in the bag
remove(item) Remove and return item from the bag or raise exception if it's not in the bag
iterator() Create and return iterator to iterate over the bag

# Assertions For Testable Code

#### What is an assertion?

An assertion is an executable check for a property that must be true of the code.

#### Rule 1

Assertions Aren't For Error Handling (that's what exceptions are for)
• Don't assert anything about the result of another entity's logic

#### Rule 2

No Side Effects
• Assertions shouldn't change the state of any variables outside the scope of the assertion

#### Rule 3

No Silly Assertions
• Check for errors in your own logic
• Don't check if the user did something wrong (in the assertion)
• Don't check that any other entity is working correctly
From Udacity's Software Testing course notes.

# Creating Testable Software

1. Create Clean Code -- code for which you can:
• describe exactly what it does
• describe exactly how it interacts with other entities
from unittest import TestCaseimport sysfrom random import randintfrom nose.tools import raisesfrom softwaretesting import fixedsizequeueMAXINT = sys.maxintMININT = -sys.maxint - 1class TestFixedSizeQueue(TestCase):def setUp(self):self.size = 10self.items = [randint(MININT, MAXINT) for item in range(self.size)]self.q = fixedsizequeue.FixedsizeQueue(self.size)returndef test_enqueue(self):"""        The enqueue takes a single integer and returns True if added.        """self.q.reset()self.assertFalse(self.q.full())for index, item in enumerate(self.items):self.assertTrue(self.q.enqueue(item))self.assertEqual(index + 1, self.q.size)self.assertEqual(item, self.q.data[index])self.assertEqual((index + 1) % len(self.items), self.q.tail)self.assertFalse(self.q.empty())self.assertTrue(self.q.full())returndef test_dequeue(self):"""        The dequeue returns the oldest item or None.        """self.q.reset()self.assertIsNone(self.q.dequeue())self.assertEqual(0, self.q.size)for item in self.items:self.q.enqueue(item)for expected in self.items:actual = self.q.dequeue()self.assertEqual(expected, actual)self.assertTrue(self.q.empty())self.assertEqual(0, self.q.head)returndef test_reset(self):"""        The reset resets the pointers and count        """self.q.enqueue(9)self.q.enqueue(8)self.q.dequeue()self.q.reset()self.assertTrue(self.q.empty())self.assertEqual(0, self.q.size)self.assertEqual(0, self.q.tail)self.assertEqual(0, self.q.head)returndef test_empty(self):"""        If the queue is empty dequeue returns False and does nothing.        """q = fixedsizequeue.FixedsizeQueue(3)self.assertTrue(q.empty())q.enqueue(randint(MININT, MAXINT))self.assertFalse(q.empty())q.dequeue()self.assertTrue(q.empty())returndef test_overfull(self):"""        The array doesn't add items once full        """self.q.reset()for item in self.items:self.q.enqueue(item)self.assertEqual(0, self.q.tail)self.assertTrue(self.q.full())self.assertEqual(len(self.items), self.q.size)self.assertFalse(self.q.enqueue(1))self.assertEqual(len(self.items), self.q.size)self.assertEqual(self.q.size, self.q.max)for item in self.items:self.assertEqual(item, self.q.dequeue())self.assertTrue(self.q.empty())return@raises(TypeError)def test_non_int(self):"""        The array has to be homogeneous.        """self.q.enqueue(1.0)return# end class TestFixedSizeQueue