How Can I Repeat Each Test Multiple Times In A Py.test Run?

I want to run each selected py.test item an arbitrary number of times, sequentially. I don't see any standard py.test mechanism for doing this. I attempted to do this in the pytest

Solution 1:

One possible strategy is parameterizing the test in question, but not explicitly using the parameter.

For example:

@pytest.mark.parametrize('execution_number', range(5))defrun_multiple_times(execution_number):
    assertTrue

The above test should run five times.

Check out the parametrization documentation: https://pytest.org/latest/parametrize.html

Solution 2:

The pytest module pytest-repeat exists for this purpose, and I recommend using modules where possible, rather than re-implementing their functionality yourself.

To use it simply add pytest-repeat to your requirements.txt or pip install pytest-repeat, then execute your tests with --count n.

Solution 3:

In order to run each test a number of times, we will programmatically parameterize each test as the tests are being generated.

First, let's add the parser option (include the following in one of your conftest.py's):

defpytest_addoption(parser):
    parser.addoption('--repeat', action='store',
        help='Number of times to repeat each test')

Now we add a "pytest_generate_tests" hook. Here is where the magic happens.

defpytest_generate_tests(metafunc):
    if metafunc.config.option.repeat isnotNone:
        count = int(metafunc.config.option.repeat)

        # We're going to duplicate these tests by parametrizing them,# which requires that each test has a fixture to accept the parameter.# We can add a new fixture like so:
        metafunc.fixturenames.append('tmp_ct')

        # Now we parametrize. This is what happens when we do e.g.,# @pytest.mark.parametrize('tmp_ct', range(count))# def test_foo(): pass
        metafunc.parametrize('tmp_ct', range(count))

Running without the repeat flag:

(env) $ py.test test.py -vv
============================= test session starts ==============================
platform darwin -- Python 2.7.5 -- py-1.4.20 -- pytest-2.5.2 -- env/bin/python
collected 2 items 

test.py:4: test_1 PASSED
test.py:8: test_2 PASSED

=========================== 2 passed in 0.01 seconds ===========================

Running with the repeat flag:

(env) $ py.test test.py -vv --repeat 3
============================= test session starts ==============================
platform darwin -- Python 2.7.5 -- py-1.4.20 -- pytest-2.5.2 -- env/bin/python
collected 6 items 

test.py:4: test_1[0] PASSEDtest.py:4: test_1[1] PASSEDtest.py:4: test_1[2] PASSEDtest.py:8: test_2[0] PASSEDtest.py:8: test_2[1] PASSEDtest.py:8: test_2[2] PASSED

=========================== 6 passed in 0.01 seconds ===========================

Further reading:

• https://pytest.org/latest/plugins.html#well-specified-hooks
• https://pytest.org/latest/example/parametrize.html

Solution 4:

Based on Frank T's suggestion, I found a very simple solution in the pytest_generate_tests() callout:

parser.addoption ('--count', default=1, type='int', metavar='count', help='Run each test the specified number of times')

defpytest_generate_tests (metafunc):
    for i inrange (metafunc.config.option.count):
        metafunc.addcall()

Now executing py.test --count 5 causes each test to be executed five times in the test session.

And it requires no changes to any of our existing tests.

Solution 5:

Based on what I've seen here, and given that I already do some filtering of tests in pytest_collection_modifyitems, my method of choice is the following. In conftest.py

defpytest_addoption(parser):
    parser.addoption ('--count', default=1, type='int', metavar='count', help='Run each test the specified number of times')


defpytest_collection_modifyitems(session, config, items):
    count = config.option.count
    items[:] = items * count  # add each test multiple times