forked from test_framework/pytest-bdd
1282 lines
40 KiB
ReStructuredText
1282 lines
40 KiB
ReStructuredText
BDD library for the py.test runner
|
||
==================================
|
||
|
||
.. image:: http://img.shields.io/pypi/v/pytest-bdd.svg
|
||
:target: https://pypi.python.org/pypi/pytest-bdd
|
||
.. image:: http://img.shields.io/coveralls/pytest-dev/pytest-bdd/master.svg
|
||
:target: https://coveralls.io/r/pytest-dev/pytest-bdd
|
||
.. image:: https://travis-ci.org/pytest-dev/pytest-bdd.svg?branch=master
|
||
:target: https://travis-ci.org/pytest-dev/pytest-bdd
|
||
.. image:: https://readthedocs.org/projects/pytest-bdd/badge/?version=latest
|
||
:target: https://readthedocs.org/projects/pytest-bdd/?badge=latest
|
||
:alt: Documentation Status
|
||
|
||
pytest-bdd implements a subset of Gherkin language for the automation of the project
|
||
requirements testing and easier behavioral driven development.
|
||
|
||
Unlike many other BDD tools it doesn't require a separate runner and benefits from
|
||
the power and flexibility of the pytest. It allows to unify your unit and functional
|
||
tests, easier continuous integration server configuration and maximal reuse of the
|
||
tests setup.
|
||
|
||
Pytest fixtures written for the unit tests can be reused for the setup and actions
|
||
mentioned in the feature steps with dependency injection, which allows a true BDD
|
||
just-enough specification of the requirements without maintaining any context object
|
||
containing the side effects of the Gherkin imperative declarations.
|
||
|
||
.. _behave: https://pypi.python.org/pypi/behave
|
||
.. _pytest-splinter: https://github.com/pytest-dev/pytest-splinter
|
||
|
||
Install pytest-bdd
|
||
------------------
|
||
|
||
::
|
||
|
||
pip install pytest-bdd
|
||
|
||
|
||
The minimal required version of pytest is 2.8.1.
|
||
|
||
|
||
Example
|
||
-------
|
||
|
||
An example test for a blog hosting software could look like this.
|
||
Note that pytest-splinter_ is used to get the browser fixture.
|
||
|
||
publish_article.feature:
|
||
|
||
.. code-block:: gherkin
|
||
|
||
Feature: Blog
|
||
A site where you can publish your articles.
|
||
|
||
Scenario: Publishing the article
|
||
Given I'm an author user
|
||
And I have an article
|
||
When I go to the article page
|
||
And I press the publish button
|
||
Then I should not see the error message
|
||
And the article should be published # Note: will query the database
|
||
|
||
Note that only one feature is allowed per feature file.
|
||
|
||
test_publish_article.py:
|
||
|
||
.. code-block:: python
|
||
|
||
from pytest_bdd import scenario, given, when, then
|
||
|
||
@scenario('publish_article.feature', 'Publishing the article')
|
||
def test_publish():
|
||
pass
|
||
|
||
|
||
@given("I'm an author user")
|
||
def author_user(auth, author):
|
||
auth['user'] = author.user
|
||
|
||
|
||
@given('I have an article')
|
||
def article(author):
|
||
return create_test_article(author=author)
|
||
|
||
|
||
@when('I go to the article page')
|
||
def go_to_article(article, browser):
|
||
browser.visit(urljoin(browser.url, '/manage/articles/{0}/'.format(article.id)))
|
||
|
||
|
||
@when('I press the publish button')
|
||
def publish_article(browser):
|
||
browser.find_by_css('button[name=publish]').first.click()
|
||
|
||
|
||
@then('I should not see the error message')
|
||
def no_error_message(browser):
|
||
with pytest.raises(ElementDoesNotExist):
|
||
browser.find_by_css('.message.error').first
|
||
|
||
|
||
@then('the article should be published')
|
||
def article_is_published(article):
|
||
article.refresh() # Refresh the object in the SQLAlchemy session
|
||
assert article.is_published
|
||
|
||
|
||
Scenario decorator
|
||
------------------
|
||
|
||
Scenario decorator can accept such optional keyword arguments:
|
||
|
||
* ``encoding`` - decode content of feature file in specific encoding. UTF-8 is default.
|
||
* ``example_converters`` - mapping to pass functions to convert example values provided in feature files.
|
||
|
||
Function decorated with `scenario` decorator behaves like a normal test function,
|
||
which will be executed after all scenario steps.
|
||
You can consider it as a normal pytest test function, e.g. order fixtures there,
|
||
call other functions and make assertions:
|
||
|
||
|
||
.. code-block:: python
|
||
|
||
from pytest_bdd import scenario, given, when, then
|
||
|
||
@scenario('publish_article.feature', 'Publishing the article')
|
||
def test_publish(browser):
|
||
assert article.title in browser.html
|
||
|
||
|
||
Step aliases
|
||
------------
|
||
|
||
Sometimes it is needed to declare the same fixtures or steps with the
|
||
different names for better readability. In order to use the same step
|
||
function with multiple step names simply decorate it multiple times:
|
||
|
||
.. code-block:: python
|
||
|
||
@given('I have an article')
|
||
@given('there\'s an article')
|
||
def article(author):
|
||
return create_test_article(author=author)
|
||
|
||
Note that the given step aliases are independent and will be executed
|
||
when mentioned.
|
||
|
||
For example if you associate your resource to some owner or not. Admin
|
||
user can’t be an author of the article, but articles should have a
|
||
default author.
|
||
|
||
.. code-block:: gherkin
|
||
|
||
Scenario: I'm the author
|
||
Given I'm an author
|
||
And I have an article
|
||
|
||
|
||
Scenario: I'm the admin
|
||
Given I'm the admin
|
||
And there's an article
|
||
|
||
|
||
Given step scope
|
||
----------------
|
||
|
||
If you need your given step to be executed less than once per scenario (for example: once for module, session), you can
|
||
pass optional ``scope`` argument:
|
||
|
||
.. code-block:: python
|
||
|
||
@given('I have an article', scope='session')
|
||
def article(author):
|
||
return create_test_article(author=author)
|
||
|
||
.. code-block:: gherkin
|
||
|
||
Scenario: I'm the author
|
||
Given I'm an author
|
||
And I have an article
|
||
|
||
|
||
Scenario: I'm the admin
|
||
Given I'm the admin
|
||
And there is an article
|
||
|
||
|
||
For this example, step function for 'I have an article' given step will be executed once even though there are 2
|
||
scenarios using it.
|
||
Note that for other step types, it makes no sense to have scope larger than 'function' (the default) as they represent
|
||
an action (when step), and assertion (then step).
|
||
|
||
|
||
Step arguments
|
||
--------------
|
||
|
||
Often it's possible to reuse steps giving them a parameter(s).
|
||
This allows to have single implementation and multiple use, so less code.
|
||
Also opens the possibility to use same step twice in single scenario and with different arguments!
|
||
And even more, there are several types of step parameter parsers at your disposal
|
||
(idea taken from behave_ implementation):
|
||
|
||
.. _pypi_parse: http://pypi.python.org/pypi/parse
|
||
.. _pypi_parse_type: http://pypi.python.org/pypi/parse_type
|
||
|
||
**string** (the default)
|
||
This is the default and can be considered as a `null` or `exact` parser. It parses no parameters
|
||
and matches the step name by equality of strings.
|
||
**parse** (based on: pypi_parse_)
|
||
Provides a simple parser that replaces regular expressions for
|
||
step parameters with a readable syntax like ``{param:Type}``.
|
||
The syntax is inspired by the Python builtin ``string.format()``
|
||
function.
|
||
Step parameters must use the named fields syntax of pypi_parse_
|
||
in step definitions. The named fields are extracted,
|
||
optionally type converted and then used as step function arguments.
|
||
Supports type conversions by using type converters passed via `extra_types`
|
||
**cfparse** (extends: pypi_parse_, based on: pypi_parse_type_)
|
||
Provides an extended parser with "Cardinality Field" (CF) support.
|
||
Automatically creates missing type converters for related cardinality
|
||
as long as a type converter for cardinality=1 is provided.
|
||
Supports parse expressions like:
|
||
* ``{values:Type+}`` (cardinality=1..N, many)
|
||
* ``{values:Type*}`` (cardinality=0..N, many0)
|
||
* ``{value:Type?}`` (cardinality=0..1, optional)
|
||
Supports type conversions (as above).
|
||
**re**
|
||
This uses full regular expressions to parse the clause text. You will
|
||
need to use named groups "(?P<name>...)" to define the variables pulled
|
||
from the text and passed to your ``step()`` function.
|
||
Type conversion can only be done via `converters` step decorator argument (see example below).
|
||
|
||
The default parser is `string`, so just plain one-to-one match to the keyword definition.
|
||
Parsers except `string`, as well as their optional arguments are specified like:
|
||
|
||
for `cfparse` parser
|
||
|
||
.. code-block:: python
|
||
|
||
from pytest_bdd import parsers
|
||
|
||
@given(parsers.cfparse('there are {start:Number} cucumbers', extra_types=dict(Number=int)))
|
||
def start_cucumbers(start):
|
||
return dict(start=start, eat=0)
|
||
|
||
for `re` parser
|
||
|
||
.. code-block:: python
|
||
|
||
from pytest_bdd import parsers
|
||
|
||
@given(parsers.re(r'there are (?P<start>\d+) cucumbers'), converters=dict(start=int))
|
||
def start_cucumbers(start):
|
||
return dict(start=start, eat=0)
|
||
|
||
|
||
Example:
|
||
|
||
.. code-block:: gherkin
|
||
|
||
Scenario: Arguments for given, when, thens
|
||
Given there are 5 cucumbers
|
||
|
||
When I eat 3 cucumbers
|
||
And I eat 2 cucumbers
|
||
|
||
Then I should have 0 cucumbers
|
||
|
||
|
||
The code will look like:
|
||
|
||
.. code-block:: python
|
||
|
||
import re
|
||
from pytest_bdd import scenario, given, when, then, parsers
|
||
|
||
|
||
@scenario('arguments.feature', 'Arguments for given, when, thens')
|
||
def test_arguments():
|
||
pass
|
||
|
||
|
||
@given(parsers.parse('there are {start:d} cucumbers'))
|
||
def start_cucumbers(start):
|
||
return dict(start=start, eat=0)
|
||
|
||
|
||
@when(parsers.parse('I eat {eat:d} cucumbers'))
|
||
def eat_cucumbers(start_cucumbers, eat):
|
||
start_cucumbers['eat'] += eat
|
||
|
||
|
||
@then(parsers.parse('I should have {left:d} cucumbers'))
|
||
def should_have_left_cucumbers(start_cucumbers, start, left):
|
||
assert start_cucumbers['start'] == start
|
||
assert start - start_cucumbers['eat'] == left
|
||
|
||
Example code also shows possibility to pass argument converters which may be useful if you need to postprocess step
|
||
arguments after the parser.
|
||
|
||
You can implement your own step parser. It's interface is quite simple. The code can looks like:
|
||
|
||
.. code-block:: python
|
||
|
||
import re
|
||
|
||
from pytest_bdd import given, parsers
|
||
|
||
class MyParser(parsers.StepParser):
|
||
|
||
"""Custom parser."""
|
||
|
||
def __init__(self, name, **kwargs):
|
||
"""Compile regex."""
|
||
super(re, self).__init__(name)
|
||
self.regex = re.compile(re.sub('%(.+)%', '(?P<\1>.+)', self.name), **kwargs)
|
||
|
||
def parse_arguments(self, name):
|
||
"""Get step arguments.
|
||
|
||
:return: `dict` of step arguments
|
||
"""
|
||
return self.regex.match(name).groupdict()
|
||
|
||
def is_matching(self, name):
|
||
"""Match given name with the step name."""
|
||
return bool(self.regex.match(name))
|
||
|
||
@given(parsers.parse('there are %start% cucumbers'))
|
||
def start_cucumbers(start):
|
||
return dict(start=start, eat=0)
|
||
|
||
Step arguments are fixtures as well!
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
||
Step arguments are injected into pytest `request` context as normal fixtures with the names equal to the names of the
|
||
arguments. This opens a number of possibilies:
|
||
|
||
* you can access step's argument as a fixture in other step function just by mentioning it as an argument (just like any othe pytest fixture)
|
||
* if the name of the step argument clashes with existing fixture, it will be overridden by step's argument value; this way you can set/override the value for some fixture deeply inside of the fixture tree in a ad-hoc way by just choosing the proper name for the step argument.
|
||
|
||
|
||
Override fixtures via given steps
|
||
---------------------------------
|
||
|
||
Dependency injection is not a panacea if you have complex structure of your test setup data. Sometimes there's a need
|
||
such a given step which would imperatively change the fixture only for certain test (scenario), while for other tests
|
||
it will stay untouched. To allow this, special parameter `target_fixture` exists in the `given` decorator:
|
||
|
||
.. code-block:: python
|
||
|
||
from pytest_bdd import given
|
||
|
||
@pytest.fixture
|
||
def foo():
|
||
return "foo"
|
||
|
||
|
||
@given("I have injecting given", target_fixture="foo")
|
||
def injecting_given():
|
||
return "injected foo"
|
||
|
||
|
||
@then('foo should be "injected foo"')
|
||
def foo_is_foo(foo):
|
||
assert foo == 'injected foo'
|
||
|
||
|
||
.. code-block:: gherkin
|
||
|
||
Scenario: Test given fixture injection
|
||
Given I have injecting given
|
||
Then foo should be "injected foo"
|
||
|
||
In this example existing fixture `foo` will be overridden by given step `I have injecting given` only for scenario it's
|
||
used in.
|
||
|
||
|
||
Multiline steps
|
||
---------------
|
||
|
||
As Gherkin, pytest-bdd supports multiline steps
|
||
(aka `PyStrings <http://docs.behat.org/guides/1.gherkin.html#pystrings>`_).
|
||
But in much cleaner and powerful way:
|
||
|
||
.. code-block:: gherkin
|
||
|
||
Scenario: Multiline step using sub indentation
|
||
Given I have a step with:
|
||
Some
|
||
Extra
|
||
Lines
|
||
Then the text should be parsed with correct indentation
|
||
|
||
Step is considered as multiline one, if the **next** line(s) after it's first line, is indented relatively
|
||
to the first line. The step name is then simply extended by adding further lines with newlines.
|
||
In the example above, the Given step name will be:
|
||
|
||
.. code-block:: python
|
||
|
||
'I have a step with:\nSome\nExtra\nLines'
|
||
|
||
You can of course register step using full name (including the newlines), but it seems more practical to use
|
||
step arguments and capture lines after first line (or some subset of them) into the argument:
|
||
|
||
.. code-block:: python
|
||
|
||
import re
|
||
|
||
from pytest_bdd import given, then, scenario
|
||
|
||
|
||
@scenario(
|
||
'multiline.feature',
|
||
'Multiline step using sub indentation',
|
||
)
|
||
def test_multiline():
|
||
pass
|
||
|
||
|
||
@given(parsers.parse('I have a step with:\n{text}'))
|
||
def i_have_text(text):
|
||
return text
|
||
|
||
|
||
@then('the text should be parsed with correct indentation')
|
||
def text_should_be_correct(i_have_text, text):
|
||
assert i_have_text == text == 'Some\nExtra\nLines'
|
||
|
||
Note that `then` step definition (`text_should_be_correct`) in this example uses `text` fixture which is provided
|
||
by a a `given` step (`i_have_text`) argument with the same name (`text`). This possibility is described in
|
||
the `Step arguments are fixtures as well!`_ section.
|
||
|
||
|
||
Scenarios shortcut
|
||
------------------
|
||
|
||
If you have relatively large set of feature files, it's boring to manually bind scenarios to the tests using the
|
||
scenario decorator. Of course with the manual approach you get all the power to be able to additionally parametrize
|
||
the test, give the test function a nice name, document it, etc, but in the majority of the cases you don't need that.
|
||
Instead you want to bind `all` scenarios found in the `feature` folder(s) recursively automatically.
|
||
For this - there's a `scenarios` helper.
|
||
|
||
.. code-block:: python
|
||
|
||
from pytest_bdd import scenarios
|
||
|
||
# assume 'features' subfolder is in this file's directory
|
||
scenarios('features')
|
||
|
||
That's all you need to do to bind all scenarios found in the `features` folder!
|
||
Note that you can pass multiple paths, and those paths can be either feature files or feature folders.
|
||
|
||
|
||
.. code-block:: python
|
||
|
||
from pytest_bdd import scenarios
|
||
|
||
# pass multiple paths/files
|
||
scenarios('features', 'other_features/some.feature', 'some_other_features')
|
||
|
||
But what if you need to manually bind certain scenario, leaving others to be automatically bound?
|
||
Just write your scenario in a `normal` way, but ensure you do it `BEFORE` the call of `scenarios` helper.
|
||
|
||
|
||
.. code-block:: python
|
||
|
||
from pytest_bdd import scenario, scenarios
|
||
|
||
@scenario('features/some.feature', 'Test something')
|
||
def test_something():
|
||
pass
|
||
|
||
# assume 'features' subfolder is in this file's directory
|
||
scenarios('features')
|
||
|
||
In the example above `test_something` scenario binding will be kept manual, other scenarios found in the `features`
|
||
folder will be bound automatically.
|
||
|
||
|
||
Scenario outlines
|
||
-----------------
|
||
|
||
Scenarios can be parametrized to cover few cases. In Gherkin the variable
|
||
templates are written using corner braces as <somevalue>.
|
||
`Gherkin scenario outlines <http://docs.behat.org/guides/1.gherkin.html#scenario-outlines>`_ are supported by pytest-bdd
|
||
exactly as it's described in be behave_ docs.
|
||
|
||
Example:
|
||
|
||
.. code-block:: gherkin
|
||
|
||
Scenario Outline: Outlined given, when, thens
|
||
Given there are <start> cucumbers
|
||
When I eat <eat> cucumbers
|
||
Then I should have <left> cucumbers
|
||
|
||
Examples:
|
||
| start | eat | left |
|
||
| 12 | 5 | 7 |
|
||
|
||
pytest-bdd feature file format also supports example tables in different way:
|
||
|
||
|
||
.. code-block:: gherkin
|
||
|
||
Scenario Outline: Outlined given, when, thens
|
||
Given there are <start> cucumbers
|
||
When I eat <eat> cucumbers
|
||
Then I should have <left> cucumbers
|
||
|
||
Examples: Vertical
|
||
| start | 12 | 2 |
|
||
| eat | 5 | 1 |
|
||
| left | 7 | 1 |
|
||
|
||
This form allows to have tables with lots of columns keeping the maximum text width predictable without significant
|
||
readability change.
|
||
|
||
The code will look like:
|
||
|
||
.. code-block:: python
|
||
|
||
from pytest_bdd import given, when, then, scenario
|
||
|
||
|
||
@scenario(
|
||
'outline.feature',
|
||
'Outlined given, when, thens',
|
||
example_converters=dict(start=int, eat=float, left=str)
|
||
)
|
||
def test_outlined():
|
||
pass
|
||
|
||
|
||
@given('there are <start> cucumbers')
|
||
def start_cucumbers(start):
|
||
assert isinstance(start, int)
|
||
return dict(start=start)
|
||
|
||
|
||
@when('I eat <eat> cucumbers')
|
||
def eat_cucumbers(start_cucumbers, eat):
|
||
assert isinstance(eat, float)
|
||
start_cucumbers['eat'] = eat
|
||
|
||
|
||
@then('I should have <left> cucumbers')
|
||
def should_have_left_cucumbers(start_cucumbers, start, eat, left):
|
||
assert isinstance(left, str)
|
||
assert start - eat == int(left)
|
||
assert start_cucumbers['start'] == start
|
||
assert start_cucumbers['eat'] == eat
|
||
|
||
Example code also shows possibility to pass example converters which may be useful if you need parameter types
|
||
different than strings.
|
||
|
||
|
||
Feature examples
|
||
^^^^^^^^^^^^^^^^
|
||
|
||
It's possible to declare example table once for the whole feature, and it will be shared
|
||
among all the scenarios of that feature:
|
||
|
||
.. code-block:: gherkin
|
||
|
||
Feature: Outline
|
||
|
||
Examples:
|
||
| start | eat | left |
|
||
| 12 | 5 | 7 |
|
||
| 5 | 4 | 1 |
|
||
|
||
Scenario Outline: Eat cucumbers
|
||
Given there are <start> cucumbers
|
||
When I eat <eat> cucumbers
|
||
Then I should have <left> cucumbers
|
||
|
||
Scenario Outline: Eat apples
|
||
Given there are <start> apples
|
||
When I eat <eat> apples
|
||
Then I should have <left> apples
|
||
|
||
For some more complex case, you might want to parametrize on both levels: feature and scenario.
|
||
This is allowed as long as parameter names do not clash:
|
||
|
||
|
||
.. code-block:: gherkin
|
||
|
||
Feature: Outline
|
||
|
||
Examples:
|
||
| start | eat | left |
|
||
| 12 | 5 | 7 |
|
||
| 5 | 4 | 1 |
|
||
|
||
Scenario Outline: Eat fruits
|
||
Given there are <start> <fruits>
|
||
When I eat <eat> <fruits>
|
||
Then I should have <left> <fruits>
|
||
|
||
Examples:
|
||
| fruits |
|
||
| oranges |
|
||
| apples |
|
||
|
||
Scenario Outline: Eat vegetables
|
||
Given there are <start> <vegetables>
|
||
When I eat <eat> <vegetables>
|
||
Then I should have <left> <vegetables>
|
||
|
||
Examples:
|
||
| vegetables |
|
||
| carrots |
|
||
| tomatoes |
|
||
|
||
|
||
Combine scenario outline and pytest parametrization
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
||
It's also possible to parametrize the scenario on the python side.
|
||
The reason for this is that it is sometimes not needed to mention example table for every scenario.
|
||
|
||
The code will look like:
|
||
|
||
.. code-block:: python
|
||
|
||
import pytest
|
||
from pytest_bdd import scenario, given, when, then
|
||
|
||
|
||
# Here we use pytest to parametrize the test with the parameters table
|
||
@pytest.mark.parametrize(
|
||
['start', 'eat', 'left'],
|
||
[(12, 5, 7)])
|
||
@scenario(
|
||
'parametrized.feature',
|
||
'Parametrized given, when, thens',
|
||
)
|
||
# Note that we should take the same arguments in the test function that we use
|
||
# for the test parametrization either directly or indirectly (fixtures depend on them).
|
||
def test_parametrized(start, eat, left):
|
||
"""We don't need to do anything here, everything will be managed by the scenario decorator."""
|
||
|
||
|
||
@given('there are <start> cucumbers')
|
||
def start_cucumbers(start):
|
||
return dict(start=start)
|
||
|
||
|
||
@when('I eat <eat> cucumbers')
|
||
def eat_cucumbers(start_cucumbers, start, eat):
|
||
start_cucumbers['eat'] = eat
|
||
|
||
|
||
@then('I should have <left> cucumbers')
|
||
def should_have_left_cucumbers(start_cucumbers, start, eat, left):
|
||
assert start - eat == left
|
||
assert start_cucumbers['start'] == start
|
||
assert start_cucumbers['eat'] == eat
|
||
|
||
With a parametrized.feature file:
|
||
|
||
.. code-block:: gherkin
|
||
|
||
Feature: parametrized
|
||
|
||
Scenario: Parametrized given, when, thens
|
||
Given there are <start> cucumbers
|
||
When I eat <eat> cucumbers
|
||
Then I should have <left> cucumbers
|
||
|
||
|
||
The significant downside of this approach is inability to see the test table from the feature file.
|
||
|
||
|
||
Organizing your scenarios
|
||
-------------------------
|
||
|
||
The more features and scenarios you have, the more important becomes the question about their organization.
|
||
The things you can do (and that is also a recommended way):
|
||
|
||
* organize your feature files in the folders by semantic groups:
|
||
|
||
::
|
||
|
||
features
|
||
│
|
||
├──frontend
|
||
│ │
|
||
│ └──auth
|
||
│ │
|
||
│ └──login.feature
|
||
└──backend
|
||
│
|
||
└──auth
|
||
│
|
||
└──login.feature
|
||
|
||
This looks fine, but how do you run tests only for certain feature?
|
||
As pytest-bdd uses pytest, and bdd scenarios are actually normal tests. But test files
|
||
are separate from the feature files, the mapping is up to developers, so the test files structure can look
|
||
completely different:
|
||
|
||
::
|
||
|
||
tests
|
||
│
|
||
└──functional
|
||
│
|
||
└──test_auth.py
|
||
│
|
||
└ """Authentication tests."""
|
||
from pytest_bdd import scenario
|
||
|
||
@scenario('frontend/auth/login.feature')
|
||
def test_logging_in_frontend():
|
||
pass
|
||
|
||
@scenario('backend/auth/login.feature')
|
||
def test_logging_in_backend():
|
||
pass
|
||
|
||
|
||
For picking up tests to run we can use
|
||
`tests selection <http://pytest.org/latest/usage.html#specifying-tests-selecting-tests>`_ technique. The problem is that
|
||
you have to know how your tests are organized, knowing only the feature files organization is not enough.
|
||
`cucumber tags <https://github.com/cucumber/cucumber/wiki/Tags>`_ introduce standard way of categorizing your features
|
||
and scenarios, which pytest-bdd supports. For example, we could have:
|
||
|
||
.. code-block:: gherkin
|
||
|
||
@login @backend
|
||
Feature: Login
|
||
|
||
@successful
|
||
Scenario: Successful login
|
||
|
||
|
||
pytest-bdd uses `pytest markers <http://pytest.org/latest/mark.html#mark>`_ as a `storage` of the tags for the given
|
||
scenario test, so we can use standard test selection:
|
||
|
||
.. code-block:: bash
|
||
|
||
py.test -k "backend and login and successful"
|
||
|
||
The feature and scenario markers are not different from standard pytest markers, and the `@` symbol is stripped out
|
||
automatically to allow test selector expressions. If you want to have bdd-related tags to be distinguishable from the
|
||
other test markers, use prefix like `bdd`.
|
||
Note that if you use pytest `--strict` option, all bdd tags mentioned in the feature files should be also in the
|
||
`markers` setting of the `pytest.ini` config. Also for tags please use names which are python-compartible variable
|
||
names, eg starts with a non-number, underscore alphanumberic, etc. That way you can safely use tags for tests filtering.
|
||
|
||
You can customize how hooks are converted to pytest marks by implementing the
|
||
``pytest_bdd_apply_tag`` hook and returning ``True`` from it:
|
||
|
||
.. code-block:: python
|
||
|
||
def pytest_bdd_apply_tag(tag, function):
|
||
if tag == 'todo':
|
||
marker = pytest.mark.skip(reason="Not implemented yet")
|
||
marker(function)
|
||
return True
|
||
else:
|
||
# Fall back to pytest-bdd's default behavior
|
||
return None
|
||
|
||
Test setup
|
||
----------
|
||
|
||
Test setup is implemented within the Given section. Even though these steps
|
||
are executed imperatively to apply possible side-effects, pytest-bdd is trying
|
||
to benefit of the PyTest fixtures which is based on the dependency injection
|
||
and makes the setup more declarative style.
|
||
|
||
.. code-block:: python
|
||
|
||
@given('I have a beautiful article')
|
||
def article():
|
||
return Article(is_beautiful=True)
|
||
|
||
This also declares a PyTest fixture "article" and any other step can depend on it.
|
||
|
||
.. code-block:: gherkin
|
||
|
||
Given I have a beautiful article
|
||
When I publish this article
|
||
|
||
When step is referring the article to publish it.
|
||
|
||
.. code-block:: python
|
||
|
||
@when('I publish this article')
|
||
def publish_article(article):
|
||
article.publish()
|
||
|
||
Many other BDD toolkits operate a global context and put the side effects there.
|
||
This makes it very difficult to implement the steps, because the dependencies
|
||
appear only as the side-effects in the run-time and not declared in the code.
|
||
The publish article step has to trust that the article is already in the context,
|
||
has to know the name of the attribute it is stored there, the type etc.
|
||
|
||
In pytest-bdd you just declare an argument of the step function that it depends on
|
||
and the PyTest will make sure to provide it.
|
||
|
||
Still side effects can be applied in the imperative style by design of the BDD.
|
||
|
||
.. code-block:: gherkin
|
||
|
||
Given I have a beautiful article
|
||
And my article is published
|
||
|
||
Functional tests can reuse your fixture libraries created for the unit-tests and upgrade
|
||
them by applying the side effects.
|
||
|
||
.. code-block:: python
|
||
|
||
given('I have a beautiful article', fixture='article')
|
||
|
||
@given('my article is published')
|
||
def published_article(article):
|
||
article.publish()
|
||
return article
|
||
|
||
This way side-effects were applied to our article and PyTest makes sure that all
|
||
steps that require the "article" fixture will receive the same object. The value
|
||
of the "published_article" and the "article" fixtures is the same object.
|
||
|
||
Fixtures are evaluated only once within the PyTest scope and their values are cached.
|
||
In case of Given steps and the step arguments mentioning the same given step makes
|
||
no sense. It won't be executed second time.
|
||
|
||
.. code-block:: gherkin
|
||
|
||
Given I have a beautiful article
|
||
And some other thing
|
||
And I have a beautiful article # Won't be executed, exception is raised
|
||
|
||
|
||
pytest-bdd will raise an exception even in the case of the steps that use regular expression
|
||
patterns to get arguments.
|
||
|
||
|
||
.. code-block:: gherkin
|
||
|
||
Given I have 1 cucumbers
|
||
And I have 2 cucumbers # Exception is raised
|
||
|
||
Will raise an exception if the step is using the regular expression pattern.
|
||
|
||
.. code-block:: python
|
||
|
||
@given(re.compile('I have (?P<n>\d+) cucumbers'))
|
||
def cucumbers(n):
|
||
return create_cucumbers(n)
|
||
|
||
|
||
Backgrounds
|
||
-----------
|
||
|
||
It's often the case that to cover certain feature, you'll need multiple scenarios. And it's logical that the
|
||
setup for those scenarios will have some common parts (if not equal). For this, there are `backgrounds`.
|
||
pytest-bdd implements `Gherkin backgrounds <http://docs.behat.org/en/v2.5/guides/1.gherkin.html#backgrounds>`_ for
|
||
features.
|
||
|
||
.. code-block:: gherkin
|
||
|
||
Feature: Multiple site support
|
||
|
||
Background:
|
||
Given a global administrator named "Greg"
|
||
And a blog named "Greg's anti-tax rants"
|
||
And a customer named "Wilson"
|
||
And a blog named "Expensive Therapy" owned by "Wilson"
|
||
|
||
Scenario: Wilson posts to his own blog
|
||
Given I am logged in as Wilson
|
||
When I try to post to "Expensive Therapy"
|
||
Then I should see "Your article was published."
|
||
|
||
Scenario: Greg posts to a client's blog
|
||
Given I am logged in as Greg
|
||
When I try to post to "Expensive Therapy"
|
||
Then I should see "Your article was published."
|
||
|
||
In this example, all steps from the background will be executed before all the scenario's own given
|
||
steps, adding possibility to prepare some common setup for multiple scenarios in a single feature.
|
||
About background best practices, please read
|
||
`here <https://github.com/cucumber/cucumber/wiki/Background#good-practices-for-using-background>`_.
|
||
|
||
.. NOTE:: There is only step "Given" should be used in "Background" section,
|
||
steps "When" and "Then" are prohibited, because their purpose are
|
||
related to actions and consuming outcomes, that is conflict with
|
||
"Background" aim - prepare system for tests or "put the system
|
||
in a known state" as "Given" does it.
|
||
The statement above is applied for strict Gherkin mode, which is
|
||
enabled by default.
|
||
|
||
|
||
Reusing fixtures
|
||
----------------
|
||
|
||
Sometimes scenarios define new names for the existing fixture that can be
|
||
inherited (reused). For example, if we have pytest fixture:
|
||
|
||
|
||
.. code-block:: python
|
||
|
||
@pytest.fixture
|
||
def article():
|
||
"""Test article."""
|
||
return Article()
|
||
|
||
|
||
Then this fixture can be reused with other names using given():
|
||
|
||
.. code-block:: python
|
||
|
||
given('I have beautiful article', fixture='article')
|
||
|
||
This will be equivalent to:
|
||
|
||
|
||
.. code-block:: python
|
||
|
||
@given('I have beautiful article')
|
||
def i_have_an_article(article):
|
||
"""I have an article."""
|
||
return article
|
||
|
||
|
||
Reusing steps
|
||
-------------
|
||
|
||
It is possible to define some common steps in the parent conftest.py and
|
||
simply expect them in the child test file.
|
||
|
||
common_steps.feature:
|
||
|
||
.. code-block:: gherkin
|
||
|
||
Scenario: All steps are declared in the conftest
|
||
Given I have a bar
|
||
Then bar should have value "bar"
|
||
|
||
conftest.py:
|
||
|
||
.. code-block:: python
|
||
|
||
from pytest_bdd import given, then
|
||
|
||
|
||
@given('I have a bar')
|
||
def bar():
|
||
return 'bar'
|
||
|
||
|
||
@then('bar should have value "bar"')
|
||
def bar_is_bar(bar):
|
||
assert bar == 'bar'
|
||
|
||
test_common.py:
|
||
|
||
.. code-block:: python
|
||
|
||
@scenario('common_steps.feature', 'All steps are declared in the conftest')
|
||
def test_conftest():
|
||
pass
|
||
|
||
There are no definitions of the steps in the test file. They were
|
||
collected from the parent conftests.
|
||
|
||
|
||
Using unicode in the feature files
|
||
----------------------------------
|
||
|
||
As mentioned above, by default, utf-8 encoding is used for parsing feature files.
|
||
For steps definition, you can both use unicode- and bytestrings equally.
|
||
However, for argumented steps, if you need to use unicode symbols in it's regular expression, use `u` sign with regex:
|
||
|
||
|
||
.. code-block:: python
|
||
|
||
@given(re.compile(u"у мене є рядок який містить '{0}'".format('(?P<content>.+)')))
|
||
def there_is_a_string_with_content(content, string):
|
||
"""Create string with unicode content."""
|
||
string['content'] = content
|
||
|
||
|
||
Default steps
|
||
-------------
|
||
|
||
Here is the list of steps that are implemented inside of the pytest-bdd:
|
||
|
||
given
|
||
* trace - enters the `pdb` debugger via `pytest.set_trace()`
|
||
when
|
||
* trace - enters the `pdb` debugger via `pytest.set_trace()`
|
||
then
|
||
* trace - enters the `pdb` debugger via `pytest.set_trace()`
|
||
|
||
|
||
Feature file paths
|
||
------------------
|
||
|
||
By default, pytest-bdd will use current module's path as base path for finding feature files, but this behaviour can be changed in the pytest configuration file (i.e. `pytest.ini`, `tox.ini` or `setup.cfg`) by declaring the new base path in the `bdd_features_base_dir` key. The path is interpreted as relative to the working directory when starting pytest.
|
||
You can also override features base path on a per-scenario basis, in order to override the path for specific tests.
|
||
|
||
pytest.ini:
|
||
|
||
.. code-block:: ini
|
||
|
||
[pytest]
|
||
bdd_features_base_dir = features/
|
||
|
||
tests/test_publish_article.py:
|
||
|
||
.. code-block:: python
|
||
|
||
from pytest_bdd import scenario
|
||
|
||
@scenario('foo.feature', 'Foo feature in features/foo.feature')
|
||
def test_foo():
|
||
pass
|
||
|
||
@scenario(
|
||
'foo.feature',
|
||
'Foo feature in tests/local-features/foo.feature',
|
||
features_base_dir='./local-features/',
|
||
)
|
||
def test_foo_local():
|
||
pass
|
||
|
||
The `features_base_dir` parameter can also be passed to the `@scenario` decorator.
|
||
|
||
|
||
Avoid retyping the feature file name
|
||
------------------------------------
|
||
|
||
If you want to avoid retyping the feature file name when defining your scenarios in a test file, use functools.partial.
|
||
This will make your life much easier when defining multiple scenarios in a test file. For example:
|
||
|
||
test_publish_article.py:
|
||
|
||
.. code-block:: python
|
||
|
||
from functools import partial
|
||
|
||
import pytest_bdd
|
||
|
||
|
||
scenario = partial(pytest_bdd.scenario, '/path/to/publish_article.feature')
|
||
|
||
|
||
@scenario('Publishing the article')
|
||
def test_publish():
|
||
pass
|
||
|
||
|
||
@scenario('Publishing the article as unprivileged user')
|
||
def test_publish_unprivileged():
|
||
pass
|
||
|
||
|
||
You can learn more about `functools.partial <http://docs.python.org/2/library/functools.html#functools.partial>`_
|
||
in the Python docs.
|
||
|
||
|
||
Relax strict Gherkin language validation
|
||
----------------------------------------
|
||
|
||
If your scenarios are not written in `proper` Gherkin language, e.g. they are more like textual scripts, then
|
||
you might find it hard to use `pytest-bdd` as by default it validates the order of step types (given-when-then).
|
||
To relax that validation, just pass ``strict_gherkin=False`` to the ``scenario`` and ``scenarios`` decorators:
|
||
|
||
test_publish_article.py:
|
||
|
||
.. code-block:: python
|
||
|
||
from pytest_bdd import scenario
|
||
|
||
@scenario('publish_article.feature', 'Publishing the article in a weird way', strict_gherkin=False)
|
||
def test_publish():
|
||
pass
|
||
|
||
|
||
Hooks
|
||
-----
|
||
|
||
pytest-bdd exposes several `pytest hooks <http://pytest.org/latest/plugins.html#well-specified-hooks>`_
|
||
which might be helpful building useful reporting, visualization, etc on top of it:
|
||
|
||
* pytest_bdd_before_scenario(request, feature, scenario) - Called before scenario is executed
|
||
|
||
* pytest_bdd_after_scenario(request, feature, scenario) - Called after scenario is executed
|
||
(even if one of steps has failed)
|
||
|
||
* pytest_bdd_before_step(request, feature, scenario, step, step_func) - Called before step function
|
||
is executed and it's arguments evaluated
|
||
|
||
* pytest_bdd_before_step_call(request, feature, scenario, step, step_func, step_func_args) - Called before step
|
||
* function is executed with evaluated arguments
|
||
|
||
* pytest_bdd_after_step(request, feature, scenario, step, step_func, step_func_args) - Called after step function
|
||
is successfully executed
|
||
|
||
* pytest_bdd_step_error(request, feature, scenario, step, step_func, step_func_args, exception) - Called when step
|
||
function failed to execute
|
||
|
||
* pytest_bdd_step_validation_error(request, feature, scenario, step, step_func, step_func_args, exception) - Called
|
||
when step failed to validate
|
||
|
||
* pytest_bdd_step_func_lookup_error(request, feature, scenario, step, exception) - Called when step lookup failed
|
||
|
||
|
||
Browser testing
|
||
---------------
|
||
|
||
Tools recommended to use for browser testing:
|
||
|
||
* pytest-splinter_ - pytest `splinter <http://splinter.cobrateam.info/>`_ integration for the real browser testing
|
||
|
||
|
||
Reporting
|
||
---------
|
||
|
||
It's important to have nice reporting out of your bdd tests. Cucumber introduced some kind of standard for
|
||
`json format <https://www.relishapp.com/cucumber/cucumber/docs/json-output-formatter>`_
|
||
which can be used for `this <https://wiki.jenkins-ci.org/display/JENKINS/Cucumber+Test+Result+Plugin>`_ jenkins
|
||
plugin
|
||
|
||
To have an output in json format:
|
||
|
||
::
|
||
|
||
py.test --cucumberjson=<path to json report>
|
||
|
||
This will output an expanded (meaning scenario outlines will be expanded to several scenarios) cucumber format.
|
||
To also fill in parameters in the step name, you have to explicitly tell pytest-bdd to use the expanded format:
|
||
|
||
::
|
||
|
||
py.test --cucumberjson=<path to json report> --cucumberjson-expanded
|
||
|
||
To enable gherkin-formatted output on terminal, use
|
||
|
||
::
|
||
|
||
py.test --gherkin-terminal-reporter
|
||
|
||
|
||
Terminal reporter supports expanded format as well
|
||
|
||
::
|
||
|
||
py.test --gherkin-terminal-reporter-expanded
|
||
|
||
|
||
|
||
Test code generation helpers
|
||
----------------------------
|
||
|
||
For newcomers it's sometimes hard to write all needed test code without being frustrated.
|
||
To simplify their life, simple code generator was implemented. It allows to create fully functional
|
||
but of course empty tests and step definitions for given a feature file.
|
||
It's done as a separate console script provided by pytest-bdd package:
|
||
|
||
::
|
||
|
||
pytest-bdd generate <feature file name> .. <feature file nameN>
|
||
|
||
It will print the generated code to the standard output so you can easily redirect it to the file:
|
||
|
||
::
|
||
|
||
pytest-bdd generate features/some.feature > tests/functional/test_some.py
|
||
|
||
|
||
Advanced code generation
|
||
------------------------
|
||
|
||
For more experienced users, there's smart code generation/suggestion feature. It will only generate the
|
||
test code which is not yet there, checking existing tests and step definitions the same way it's done during the
|
||
test execution. The code suggestion tool is called via passing additional pytest arguments:
|
||
|
||
::
|
||
|
||
py.test --generate-missing --feature features tests/functional
|
||
|
||
The output will be like:
|
||
|
||
::
|
||
|
||
============================= test session starts ==============================
|
||
platform linux2 -- Python 2.7.6 -- py-1.4.24 -- pytest-2.6.2
|
||
plugins: xdist, pep8, cov, cache, bdd, bdd, bdd
|
||
collected 2 items
|
||
|
||
Scenario is not bound to any test: "Code is generated for scenarios which are not bound to any tests" in feature "Missing code generation" in /tmp/pytest-552/testdir/test_generate_missing0/tests/generation.feature
|
||
--------------------------------------------------------------------------------
|
||
|
||
Step is not defined: "I have a custom bar" in scenario: "Code is generated for scenario steps which are not yet defined(implemented)" in feature "Missing code generation" in /tmp/pytest-552/testdir/test_generate_missing0/tests/generation.feature
|
||
--------------------------------------------------------------------------------
|
||
Please place the code above to the test file(s):
|
||
|
||
@scenario('tests/generation.feature', 'Code is generated for scenarios which are not bound to any tests')
|
||
def test_Code_is_generated_for_scenarios_which_are_not_bound_to_any_tests():
|
||
"""Code is generated for scenarios which are not bound to any tests."""
|
||
|
||
|
||
@given('I have a custom bar')
|
||
def I_have_a_custom_bar():
|
||
"""I have a custom bar."""
|
||
|
||
As as side effect, the tool will validate the files for format errors, also some of the logic bugs, for example the
|
||
ordering of the types of the steps.
|
||
|
||
Migration of your tests from versions 2.x.x
|
||
------------------------------------------------
|
||
|
||
In version 3.0.0, the fixtures ``pytestbdd_feature_base_dir`` and ``pytestbdd_strict_gherkin`` have been removed.
|
||
|
||
If you used ``pytestbdd_feature_base_dir`` fixture to override the path discovery, you can instead configure it in ``pytest.ini``:
|
||
|
||
.. code-block:: ini
|
||
|
||
[pytest]
|
||
bdd_features_base_dir = features/
|
||
|
||
For more details, check the `Feature file paths`_ section.
|
||
|
||
If you used ``pytestbdd_strict_gherkin`` fixture to relax the parser, you can instead specify ``strict_gherking=False`` in the declaration of your scenarios, or change it globally in the pytest configuration file:
|
||
|
||
.. code-block:: ini
|
||
|
||
[pytest]
|
||
bdd_strict_gherkin = false
|
||
|
||
For more details, check the `Relax strict Gherkin language validation`_ section.
|
||
|
||
|
||
|
||
Migration of your tests from versions 0.x.x-1.x.x
|
||
-------------------------------------------------
|
||
|
||
In version 2.0.0, the backwards-incompartible change was introduced: scenario function can now only be used as a
|
||
decorator. Reasons for that:
|
||
|
||
* test code readability is much higher using normal python function syntax;
|
||
* pytest-bdd internals are much cleaner and shorter when using single approach instead of supporting two;
|
||
* after moving to parsing-on-import-time approach for feature files, it's not possible to detect whether it's a
|
||
decorator more or not, so to support it along with functional approach there needed to be special parameter
|
||
for that, which is also a backwards-incompartible change.
|
||
|
||
To help users migrate to newer version, there's migration subcommand of the `pytest-bdd` console script:
|
||
|
||
::
|
||
|
||
# run migration script
|
||
pytest-bdd migrate <your test folder>
|
||
|
||
Under the hood the script does the replacement from this:
|
||
|
||
.. code-block:: python
|
||
|
||
test_function = scenario('publish_article.feature', 'Publishing the article')
|
||
|
||
to this:
|
||
|
||
.. code-block:: python
|
||
|
||
@scenario('publish_article.feature', 'Publishing the article')
|
||
def test_function():
|
||
pass
|
||
|
||
|
||
License
|
||
-------
|
||
|
||
This software is licensed under the `MIT license <http://en.wikipedia.org/wiki/MIT_License>`_.
|
||
|
||
© 2013-2014 Oleg Pidsadnyi, Anatoly Bubenkov and others
|