SeleniumBase/help_docs/customizing_test_runs.md

Customizing test runs with pytest (or nosetests)

In addition to settings.py (which lets you customize SeleniumBase global properties) you can customize test runs from the command line:

• Choose the browser for tests to use (Default: Chrome)
• Choose betweeen pytest & nose unittest runners
• Choose whether to enter Debug Mode on failures
• Choose additional variables to pass into tests
• Change the automation speed (with Demo Mode)
• Choose whether to run tests multi-threaded
• Choose whether to retry failing tests
• Choose a BrowserStack server to run on
• Choose a Sauce Labs server to run on
• Choose a TestingBot server to run on
• Choose a Selenium Grid to connect to
• Choose a database to save results to
• Choose a proxy server to connect to

...and more!

Examples:

(These are run from the examples folder.)

pytest my_first_test.py

pytest my_first_test.py --demo_mode --browser=chrome

pytest my_first_test.py --browser=firefox

pytest test_suite.py --html=report.html

nosetests test_suite.py --report --show_report

pytest test_suite.py --headless -n 4

pytest test_suite.py --reruns 1 --reruns-delay 2

pytest test_suite.py --server=IP_ADDRESS --port=4444

pytest test_suite.py --proxy=IP_ADDRESS:PORT

pytest test_suite.py --proxy=USERNAME:PASSWORD@IP_ADDRESS:PORT

pytest test_fail.py --pdb -s

You can interchange pytest with nosetests, but using pytest is strongly recommended because developers stopped supporting nosetests. Chrome is the default browser if not specified.

(NOTE: If you're using pytest for running tests outside of the SeleniumBase repo, you'll want a copy of pytest.ini at the base of the new folder structure. If using nosetests, the same applies for setup.cfg.)

Running tests on BrowserStack's Selenium Grid, the Sauce Labs Selenium Grid, the TestingBot Selenium Grid, (or your own):

Here's how to connect to a BrowserStack Selenium Grid server for running tests:

pytest my_first_test.py --server=USERNAME:KEY@hub.browserstack.com --port=80

Here's how to connect to a Sauce Labs Selenium Grid server for running tests:

pytest my_first_test.py --server=USERNAME:KEY@ondemand.saucelabs.com --port=80

Here's how to connect to a TestingBot Selenium Grid server for running tests:

pytest my_first_test.py --server=USERNAME:KEY@hub.testingbot.com --port=80

Here's how to connect to a CrossBrowserTesting Selenium Grid server for running tests:

pytest my_first_test.py --server=USERNAME:KEY@hub.crossbrowsertesting.com --port=80

Or you can create your own Selenium Grid for test distribution. (See this ReadMe for details)

Example tests using Logging:

pytest test_suite.py --browser=chrome

(During test failures, logs and screenshots from the most recent test run will get saved to the latest_logs/ folder. Those logs will get moved to archived_logs/ if you have ARCHIVE_EXISTING_LOGS set to True in settings.py, otherwise log files with be cleaned up at the start of the next test run.)

Demo Mode:

If any test is moving too fast for your eyes to see what's going on, you can run it in Demo Mode by adding --demo_mode on the command line, which pauses the browser briefly between actions, highlights page elements being acted on, and lets you know what test assertions are happening in real time:

pytest my_first_test.py --browser=chrome --demo_mode

You can override the default wait time by either updating settings.py or by using --demo_sleep={NUM} when using Demo Mode. (NOTE: If you use --demo_sleep={NUM} without using --demo_mode, nothing will happen.)

pytest my_first_test.py --browser=chrome --demo_mode --demo_sleep=1.2

Passing additional data to tests:

If you want to pass additional data from the command line to your tests, you can use --data=STRING. Now inside your tests, you can use self.data to access that.

Running tests multithreaded:

To run Pytest multithreaded on multiple CPUs at the same time, add -n=NUM or -n NUM on the command line, where NUM is the number of CPUs you want to use.

Retrying failing tests automatically:

You can use --reruns NUM to retry failing tests that many times. Use --reruns-delay SECONDS to wait that many seconds between retries. Example:

pytest --reruns 5 --reruns-delay 1

Debugging tests:

You can use the following code snippets in your scripts to help you debug issues:

import time; time.sleep(5)  # sleep for 5 seconds (add this after the line you want to pause on)
import ipdb; ipdb.set_trace()  # waits for your command. n = next line of current method, c = continue, s = step / next executed line (will jump)
import pytest; pytest.set_trace()  # similar to ipdb, but specific to pytest

To pause an active test that throws an exception or error, add --pdb -s:

pytest my_first_test.py --browser=chrome --pdb -s

The code above will leave your browser window open in case there's a failure. (ipdb commands: 'c', 's', 'n' => continue, step, next).

Pytest Reports:

Using --html=report.html gives you a fancy report of the name specified after your test suite completes.

pytest test_suite.py --html=report.html

Nosetest Reports:

The --report option gives you a fancy report after your test suite completes.

nosetests test_suite.py --report

(NOTE: You can add --show_report to immediately display Nosetest reports after the test suite completes. Only use --show_report when running tests locally because it pauses the test run.)

Here are some other useful nosetest-specific arguments:

--logging-level=INFO  # Hide DEBUG messages, which can be overwhelming.
-x  # Stop running the tests after the first failure is reached.
-v  # Prints the full test name rather than a dot for each test.
--with-id  # If -v is also used, will number the tests for easy counting.

Using a Proxy Server:

If you wish to use a proxy server for your browser tests (Chrome and Firefox only), you can add --proxy=IP_ADDRESS:PORT as an argument on the command line.

pytest proxy_test.py --proxy=IP_ADDRESS:PORT

If the proxy server that you wish to use requires authentication, you can do the following (Chrome only):

pytest proxy_test.py --proxy=USERNAME:PASSWORD@IP_ADDRESS:PORT

To make things easier, you can add your frequently-used proxies to PROXY_LIST in proxy_list.py, and then use --proxy=KEY_FROM_PROXY_LIST to use the IP_ADDRESS:PORT of that key.

pytest proxy_test.py --proxy=proxy1