Setting up pytest

Source

What is this about?

This is a post to generate and document the pytest.ini configuration for testing.

Requirements

The Neurotic Repository

To make the neurotic code importable you need to install the package (in development mode, since it's (hopefully) always changing) using pip.

pip install --editable .

The Testing Dependencies

There are some things that need to be installed to get the code to run (like numpy and nltk) but for the testing this is what's needed.

expects
faker
pytest
pytest-bdd
pytest-mock
pytest-xdist

This list is in a text-file (tests/testing-requriements) so they can be installed by changing into the tests folder and running pip.

pip install --requirement testing-requirements

The Configuration File

Pytest can read in an INI formatted file. It has a bunch of configuration options but I haven't really explored them much. I mostly use it to set the command line arguments that I would otherwise pass in.

The Header

This is just the pytest section header so pytest knows to look here.

[pytest]

The Pytest BDD Features Base

Pytest-bdd will look for a feature file in the same folder as the test-file, unless you tell it not to, so let's tell it to look in the sub-folder named "features" instead.

bdd_features_base_dir = features/

The Regular Testing Run

addopts = --exitfirst --failed-first --color yes --gherkin-terminal-reporter
          --looponfail --numprocesses auto

I didn't realize it before, but as this stackoverflow answer mentions, you can break up long line in INI files (assuming that the code is using python's ConfigParser) by indenting the continuation lines (although it appears, unfortunately, to break pygments' syntax highlighting).

Here's a breakdown of the options.

  • exitfirst: stop testing after the first failed test
  • failed-first: when re-starting the tests, start with the test that failed
  • color yes: Add some highlight-coloring to the pytest output
  • gherkin-terminal-reporter: Format the output using pytest-bdd
  • looponfail: re-run the testing if a file changes
  • numprocesses auto: Run the tests in parallel across all available cores.

The numprocesses option can alternatively take logical as an argument, meaning use all logical CPUs, not physical cores (this requires you install pytest-xdist[psutil]) or the actual number of processes you want to run in parallel.

The Run-Once

This removes the automatic re-running of tests. I don't really use this, but sometimes it can be helpful, especially when using something like Selenium that slows everything down a lot, to not re-run the test every time you save a file.

# addopts = --exitfirst --failed-first --color yes --gherkin-terminal-reporter --numprocesses auto

The PUDB Run

pytest will grab standard out by default, making it impossible to run an interactive debugger. They have support for python's pdb debugger, but I use pudb instead, so this set of arguments turns off the capturing of standard out which will let you run the tests with PUDB. There is a project that integrates pudb into pytest, but it appears to have died out, so I'll just stick to my old way of doing it.

# addopts = --exitfirst --failed-first --color yes --gherkin-terminal-reporter --capture=no

Links