Such: a Functional-Test Friendly DSL

Note

New in version 0.4

Such is a DSL for writing tests with expensive, nested fixtures – which typically means functional tests. It requires the layers plugin (see Organizing Test Fixtures into Layers).

What does it look like?

Unlike some python testing DSLs, such is just plain old python.

import unittest

from nose2.tools import such


class SomeLayer:
    @classmethod
    def setUp(cls):
        it.somelayer = True

    @classmethod
    def tearDown(cls):
        del it.somelayer


#
# Such tests start with a declaration about the system under test
# and will typically bind the test declaration to a variable with
# a name that makes nice sentences, like 'this' or 'it'.
#
with such.A("system with complex setup") as it:
    #
    # Each layer of tests can define setup and teardown methods.
    # setup and teardown methods defined here run around the entire
    # group of tests, not each individual test.
    #
    @it.has_setup
    def setup():
        it.things = [1]

    @it.has_teardown
    def teardown():
        it.things = []

    #
    # The 'should' decorator is used to mark tests.
    #
    @it.should("do something")
    def test():
        assert it.things
        #
        # Tests can use all of the normal unittest TestCase assert
        # methods by calling them on the test declaration.
        #
        it.assertEqual(len(it.things), 1)

    #
    # The 'having' context manager is used to introduce a new layer,
    # one that depends on the layer(s) above it. Tests in this
    # new layer inherit all of the fixtures of the layer above.
    #
    with it.having("an expensive fixture"):

        @it.has_setup
        def setup():  # noqa: F811
            it.things.append(2)

        #
        # Tests that take an argument will be passed the
        # unittest.TestCase instance that is generated to wrap
        # them. Tests can call any and all TestCase methods on this
        # instance.
        #
        @it.should("do more things")
        def test(case):  # noqa: F811
            case.assertEqual(it.things[-1], 2)

        #
        # Layers can be nested to any depth.
        #
        with it.having("another precondition"):

            @it.has_setup
            def setup():  # noqa: F811
                it.things.append(3)

            @it.has_teardown
            def teardown():  # noqa: F811
                it.things.pop()

            @it.should("do that not this")  # type: ignore[no-redef]
            def test(case):  # noqa: F811
                it.things.append(4)
                #
                # Tests can add their own cleanup functions.
                #
                case.addCleanup(it.things.pop)
                case.assertEqual(it.things[-1], 4, it.things)

            @it.should("do this not that")  # type: ignore[no-redef]
            def test(case):  # noqa: F811
                case.assertEqual(it.things[-1], 3, it.things[:])

        #
        # A layer may have any number of sub-layers.
        #
        with it.having("a different precondition"):
            #
            # A layer defined with ``having`` can make use of
            # layers defined elsewhere. An external layer
            # pulled in with ``it.uses`` becomes a parent
            # of the current layer (though it doesn't actually
            # get injected into the layer's MRO).
            #
            it.uses(SomeLayer)

            @it.has_setup
            def setup():  # noqa: F811
                it.things.append(99)

            @it.has_teardown
            def teardown():  # noqa: F811
                it.things.pop()

            #
            # Layers can define setup and teardown methods that wrap
            # each test case, as well, corresponding to TestCase.setUp
            # and TestCase.tearDown.
            #
            @it.has_test_setup
            def test_setup(case):
                it.is_funny = True
                case.is_funny = True

            @it.has_test_teardown
            def test_teardown(case):
                delattr(it, "is_funny")
                delattr(case, "is_funny")

            @it.should("do something else")  # type: ignore[no-redef]
            def test(case):  # noqa: F811
                assert it.things[-1] == 99
                assert it.is_funny
                assert case.is_funny

            @it.should("have another test")  # type: ignore[no-redef]
            def test(case):  # noqa: F811
                assert it.is_funny
                assert case.is_funny

            @it.should("have access to an external fixture")  # type: ignore[no-redef]
            def test(case):  # noqa: F811
                assert it.somelayer

            with it.having("a case inside the external fixture"):

                @it.should("still have access to that fixture")  # type: ignore[no-redef]  # noqa: E501
                def test(case):  # noqa: F811
                    assert it.somelayer


#
# To convert the layer definitions into test cases, you have to call
# `createTests` and pass in the module globals, so that the test cases
# and layer objects can be inserted into the module.
#
it.createTests(globals())


#
# Such tests and normal tests can coexist in the same modules.
#
class NormalTest(unittest.TestCase):
    def test(self):
        pass

The tests it defines are unittest tests, and can be used with nose2 with just the layers plugin. You also have the option of activating a reporting plugin (nose2.plugins.layers.LayerReporter) to provide a more discursive brand of output:

test (test_such.NormalTest) ... ok
A system with complex setup
  should do something ... ok
  having an expensive fixture
    should do more things ... ok
    having another precondition
      should do that not this ... ok
      should do this not that ... ok
    having a different precondition
      should do something else ... ok
      should have another test ... ok

----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK

How does it work?

Such uses the things in python that are most like anonymous code blocks to allow you to construct tests with meaningful names and deeply-nested fixtures. Compared to DSLs in languages that do allow blocks, it is a little bit more verbose – the block-like decorators that mark fixture methods and test cases need to decorate something, so each fixture and test case has to have a function definition. You can use the same function name over and over here, or give each function a meaningful name.

The set of tests begins with a description of the system under test as a whole, marked with the A context manager:

from nose2.tools import such

with such.A('system described here') as it:
    # ...

Groups of tests are marked by the having context manager:

with it.having('a description of a group'):
    # ...

Within a test group (including the top-level group), fixtures are marked with decorators:

@it.has_setup
def setup():
    # ...

@it.has_test_setup
def setup_each_test_case():
    # ...

And tests are likewise marked with the should decorator:

@it.should('exhibit the behavior described here')
def test(case):
    # ...

Test cases may optionally take one argument. If they do, they will be passed the unittest.TestCase instance generated for the test. They can use this TestCase instance to execute assert methods, among other things. Test functions can also call assert methods on the top-level scenario instance, if they don’t take the case argument:

@it.should("be able to use the scenario's assert methods")
def test():
    it.assertEqual(something, 'a value')

@it.should("optionally take an argument")
def test(case):
    case.assertEqual(case.attribute, 'some value')

Finally, to actually generate tests, you must call createTests on the top-level scenario instance:

it.createTests(globals())

This call generates the unittest.TestCase instances for all of the tests, and the layer classes that hold the fixtures defined in the test groups. See Organizing Test Fixtures into Layers for more about test layers.

Running tests

Since order is often significant in functional tests, such DSL tests always execute in the order in which they are defined in the module. Parent groups run before child groups, and sibling groups and sibling tests within a group execute in the order in which they are defined.

Otherwise, tests written in the such DSL are collected and run just like any other tests, with one exception: their names. The name of a such test case is the name of its immediately surrounding group, plus the description of the test, prepended with test ####:, where #### is the test’s (0 -indexed) position within its group.

To run a case individually, you must pass in this full name – usually you’ll have to quote it. For example, to run the case should do more things defined above (assuming the layers plugin is activated by a config file, and the test module is in the normal path of test collection), you would run nose2 like this:

nose2 "test_such.having an expensive fixture.test 0000: should do more things"

That is, for the generated test case, the group description is the class name, and the test case description is the test case name. As you can see if you run an individual test with the layer reporter active, all of the group fixtures execute in proper order when a test is run individually:

$ nose2 "test_such.having an expensive fixture.test 0000: should do more things"
A system with complex setup
  having an expensive fixture
    should do more things ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

Reference

nose2.tools.such.A(description)[source]

Test scenario context manager.

Returns a nose2.tools.such.Scenario instance, which by convention is bound to it:

with such.A('test scenario') as it:
    # tests and fixtures

class nose2.tools.such.Scenario(description)[source]

A test scenario.

A test scenario defines a set of fixtures and tests that depend on those fixtures.

createTests(mod)[source]

Generate test cases for this scenario.

Warning

You must call this, passing in globals(), to generate tests from the scenario. If you don’t, no tests will be created.

it.createTests(globals())

has_setup(func)[source]

Add a setup() method to this group.

The setup() method will run once, before any of the tests in the containing group.

A group may define any number of setup() functions. They will execute in the order in which they are defined.

@it.has_setup
def setup():
    # ...

has_teardown(func)[source]

Add a teardown() method to this group.

The teardown() method will run once, after all of the tests in the containing group.

A group may define any number of teardown() functions. They will execute in the order in which they are defined.

@it.has_teardown
def teardown():
    # ...

has_test_setup(func)[source]

Add a test case setup() method to this group.

The setup() method will run before each of the tests in the containing group.

A group may define any number of test case setup() functions. They will execute in the order in which they are defined.

Test setup() functions may optionally take one argument. If they do, they will be passed the unittest.TestCase instance generated for the test.

@it.has_test_setup
def setup(case):
    # ...

has_test_teardown(func)[source]

Add a test case teardown() method to this group.

The teardown() method will run before each of the tests in the containing group.

A group may define any number of test case teardown() functions. They will execute in the order in which they are defined.

Test teardown() functions may optionally take one argument. If they do, they will be passed the unittest.TestCase instance generated for the test.

@it.has_test_teardown
def teardown(case):
    # ...

having(description)[source]

Define a new group under the current group.

Fixtures and tests defined within the block will belong to the new group.

with it.having('a description of this group'):
    # ...

should(desc)[source]

Define a test case.

Each function marked with this decorator becomes a test case in the current group.

The decorator takes one optional argument, the description of the test case: what it should do. If this argument is not provided, the docstring of the decorated function will be used as the test case description.

Test functions may optionally take one argument. If they do, they will be passed the unittest.TestCase instance generated for the test. They can use this TestCase instance to execute assert methods, among other things.

@it.should('do this')
def dothis(case):
    # ....

@it.should
def dothat():
    "do that also"
    # ....