test_framework
/
SeleniumBase
mirror of https://github.com/seleniumbase/SeleniumBase.git
3
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Refactoring command line options and docs

This commit is contained in:
Michael Mintz 2020-08-08 01:04:44 -04:00
parent c1907a0c02
commit 3a26df6546
3 changed files with 96 additions and 76 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

98
help_docs/customizing_test_runs.md
View File

@ -14,10 +14,13 @@ pytest test_swag_labs.py
pytest test_swag_labs.py --browser=firefox pytest test_swag_labs.py --browser=firefox
# Run a test in Demo Mode (highlight assertions) # Run a test in Demo Mode (highlight assertions)
pytest test_swag_labs.py --demo pytest my_first_test.py --demo
# Run another test in Demo Mode
pytest test_demo_site.py --demo
# Run a test in Headless Mode (invisible browser) # Run a test in Headless Mode (invisible browser)
pytest test_swag_labs.py --headless pytest my_first_test.py --headless
# Run tests multi-threaded using [n] threads # Run tests multi-threaded using [n] threads
pytest test_suite.py -n=4 pytest test_suite.py -n=4
@ -26,12 +29,12 @@ pytest test_suite.py -n=4
pytest test_suite.py --html=report.html pytest test_suite.py --html=report.html
# Enter Debug Mode on failures # Enter Debug Mode on failures
pytest test_fail.py --pdb -s pytest test_fail.py --pdb
# Rerun failing tests more times # Rerun failing tests more times
pytest test_suite.py --reruns=1 pytest test_suite.py --reruns=1
# Pass extra data into tests (retrieve: self.data) # Pass extra data into tests (retrieve by calling self.data)
pytest my_first_test.py --data="ABC,DEF" pytest my_first_test.py --data="ABC,DEF"
# Run tests on a local Selenium Grid # Run tests on a local Selenium Grid
@ -68,14 +71,15 @@ pytest test_swag_labs.py --mobile --metrics="411,731,3"
pytest my_first_test.py --settings-file=custom_settings.py pytest my_first_test.py --settings-file=custom_settings.py
``` ```
You can interchange **pytest** with **nosetests** for most things, but using pytest is strongly recommended because developers stopped supporting nosetests. Chrome is the default browser if not specified. You can interchange ``pytest`` with ``nosetests`` for most tests, but using ``pytest`` is recommended. (``chrome`` is the default browser if not specified.)
(NOTE: If you're using **pytest** for running tests outside of the SeleniumBase repo, **you'll want a copy of [pytest.ini](https://github.com/seleniumbase/SeleniumBase/blob/master/pytest.ini) at the base of the new folder structure**. If using **nosetests**, the same applies for [setup.cfg](https://github.com/seleniumbase/SeleniumBase/blob/master/setup.cfg).) If you're using ``pytest`` for running tests outside of the SeleniumBase repo, you'll want a copy of [pytest.ini](https://github.com/seleniumbase/SeleniumBase/blob/master/pytest.ini) at the base of the new folder structure. If using ``nosetests``, the same applies for [setup.cfg](https://github.com/seleniumbase/SeleniumBase/blob/master/setup.cfg).
Here are some useful command-line options that come with ``pytest``:
Here are some useful command-line options that come with Pytest:
```bash ```bash
-v # Prints the full test name for each test. -v # Verbose mode. Print the full name of each test run.
-q # Prints fewer details in the console output when running tests. -q # Quiet mode. Print fewer details in the console output when running tests.
-x # Stop running the tests after the first failure is reached. -x # Stop running the tests after the first failure is reached.
--html=report.html # Creates a detailed pytest-html report after tests finish. --html=report.html # Creates a detailed pytest-html report after tests finish.
--collect-only # Show what tests would get run without actually running them. --collect-only # Show what tests would get run without actually running them.
@ -86,56 +90,70 @@ Here are some useful command-line options that come with Pytest:
-m=MARKER # Only run tests that are marked with the specified pytest marker. -m=MARKER # Only run tests that are marked with the specified pytest marker.
``` ```
SeleniumBase provides additional Pytest command-line options for tests: SeleniumBase provides additional ``pytest`` command-line options for tests:
```bash ```bash
--browser=BROWSER # (The web browser to use.) --browser=BROWSER # (The web browser to use. Default: "chrome")
--cap-file=FILE # (The web browser's desired capabilities to use.) --cap-file=FILE # (The web browser's desired capabilities to use.)
--cap-string=STRING # (The web browser's desired capabilities to use.) --cap-string=STRING # (The web browser's desired capabilities to use.)
--settings-file=FILE # (Overrides SeleniumBase settings.py values.) --settings-file=FILE # (Override default SeleniumBase settings.)
--env=ENV # (Set a test environment. Use "self.env" to use this in tests.) --env=ENV # (Set a test environment. Use "self.env" to use this in tests.)
--data=DATA # (Extra data to pass to tests. Use "self.data" in tests.) --data=DATA # (Extra test data. Access with "self.data" in tests.)
--var1=DATA # (Extra data to pass to tests. Use "self.var1" in tests.) --var1=DATA # (Extra test data. Access with "self.var1" in tests.)
--var2=DATA # (Extra data to pass to tests. Use "self.var2" in tests.) --var2=DATA # (Extra test data. Access with "self.var2" in tests.)
--var3=DATA # (Extra data to pass to tests. Use "self.var3" in tests.) --var3=DATA # (Extra test data. Access with "self.var3" in tests.)
--user-data-dir=DIR # (Set the Chrome user data directory to use.) --user-data-dir=DIR # (Set the Chrome user data directory to use.)
--server=SERVER # (The server / IP address used by the tests.) --server=SERVER # (The server / IP address used by the tests.)
--port=PORT # (The port that's used by the test server.) --port=PORT # (The port that's used by the test server.)
--proxy=SERVER:PORT # (This is the proxy server:port combo used by tests.) --proxy=SERVER:PORT # (This is the proxy server:port combo used by tests.)
--agent=STRING # (This designates the web browser's User Agent to use.) --agent=STRING # (Modify the web browser's User-Agent string.)
--mobile # (The option to use the mobile emulator while running tests.) --mobile # (Use the mobile device emulator while running tests.)
--metrics=STRING # ("CSSWidth,Height,PixelRatio" for mobile emulator tests.) --metrics=STRING # (Set mobile "CSSWidth,CSSHeight,PixelRatio".)
--extension-zip=ZIP # (Load a Chrome Extension .zip file, comma-separated.) --extension-zip=ZIP # (Load a Chrome Extension .zip file, comma-separated.)
--extension-dir=DIR # (Load a Chrome Extension directory, comma-separated.) --extension-dir=DIR # (Load a Chrome Extension directory, comma-separated.)
--headless # (The option to run tests headlessly. The default on Linux OS.) --headless # (Run tests headlessly. Default mode on Linux OS.)
--headed # (The option to run tests with a GUI on Linux OS.) --headed # (Run tests with a GUI on Linux OS.)
--start-page=URL # (The starting URL for the web browser when tests begin.) --start-page=URL # (The starting URL for the web browser when tests begin.)
--archive-logs # (Archive old log files instead of deleting them.) --archive-logs # (Archive old log files instead of deleting them.)
--time-limit=SECONDS # (Safely fail any test that exceeds the limit limit.) --time-limit=SECONDS # (Safely fail any test that exceeds the limit limit.)
--slow # (The option to slow down the automation.) --slow # (Slow down the automation. Faster than using Demo Mode.)
--demo # (The option to visually see test actions as they occur.) --demo # (Slow down and visually see test actions as they occur.)
--demo-sleep=SECONDS # (The option to wait longer after Demo Mode actions.) --demo-sleep=SECONDS # (Set the wait time after Demo Mode actions.)
--highlights=NUM # (Number of highlight animations for Demo Mode actions.) --highlights=NUM # (Number of highlight animations for Demo Mode actions.)
--message-duration=SECONDS # (The time length for Messenger alerts.) --message-duration=SECONDS # (The time length for Messenger alerts.)
--check-js # (The option to check for JavaScript errors after page loads.) --check-js # (Check for JavaScript errors after page loads.)
--ad-block # (The option to block some display ads after page loads.) --ad-block # (Block some types of display ads after page loads.)
--block-images # (The option to block images from loading during tests.) --block-images # (Block images from loading during tests.)
--verify-delay=SECONDS # (The delay before MasterQA verification checks.) --verify-delay=SECONDS # (The delay before MasterQA verification checks.)
--disable-csp # (This disables the Content Security Policy of websites.) --disable-csp # (This disables the Content Security Policy of websites.)
--enable-sync # (The option to enable "Chrome Sync".) --enable-sync # (Enable "Chrome Sync".)
--use-auto-ext # (The option to use Chrome's automation extension.) --use-auto-ext # (Use Chrome's automation extension.)
--swiftshader # (The option to use Chrome's "--use-gl=swiftshader" feature.) --swiftshader # (Use Chrome's "--use-gl=swiftshader" feature.)
--incognito # (The option to enable Chrome's Incognito mode.) --incognito # (Enable Chrome's Incognito mode.)
--guest # (The option to enable Chrome's Guest mode.) --guest # (Enable Chrome's Guest mode.)
--devtools # (The option to open Chrome's DevTools when the browser opens.) --devtools # (Open Chrome's DevTools when the browser opens.)
--reuse-session # (The option to reuse the browser session between tests.) --reuse-session # (Reuse the browser session between tests.)
--crumbs # (Option to delete all cookies between tests reusing a session.) --crumbs # (Delete all cookies between tests reusing a session.)
--maximize-window # (The option to start with the web browser maximized.) --maximize-window # (Start tests with the web browser window maximized.)
--save-screenshot # (The option to save a screenshot after each test.) --save-screenshot # (Save a screenshot at the end of each test.)
--visual-baseline # (Set the visual baseline for Visual/Layout tests.) --visual-baseline # (Set the visual baseline for Visual/Layout tests.)
--timeout-multiplier=MULTIPLIER # (Multiplies the default timeout values.) --timeout-multiplier=MULTIPLIER # (Multiplies the default timeout values.)
``` ```
(For more details, see the full list of command-line options **[here](https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/plugins/pytest_plugin.py)**.) (For more details, see the full list of command-line options **[here](https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/plugins/pytest_plugin.py)**.)
You can also view a list of popular ``pytest`` options for SeleniumBase by typing:
```bash
seleniumbase options
```
Or the short form:
```bash
sbase options
```
### <img src="https://seleniumbase.io/img/sb_icon.png" title="SeleniumBase" width="30" /> Customizing default settings: ### <img src="https://seleniumbase.io/img/sb_icon.png" title="SeleniumBase" width="30" /> Customizing default settings:
An easy way to override [seleniumbase/config/settings.py](https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/config/settings.py) is by using a custom settings file. An easy way to override [seleniumbase/config/settings.py](https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/config/settings.py) is by using a custom settings file.
@ -181,6 +199,7 @@ Or you can create your own Selenium Grid for test distribution. ([See this ReadM
```bash ```bash
pytest test_suite.py --browser=chrome pytest test_suite.py --browser=chrome
``` ```
(During test failures, logs and screenshots from the most recent test run will get saved to the ``latest_logs/`` folder. Those logs will get moved to ``archived_logs/`` if you have ARCHIVE_EXISTING_LOGS set to True in [settings.py](https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/config/settings.py), otherwise log files with be cleaned up at the start of the next test run.) (During test failures, logs and screenshots from the most recent test run will get saved to the ``latest_logs/`` folder. Those logs will get moved to ``archived_logs/`` if you have ARCHIVE_EXISTING_LOGS set to True in [settings.py](https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/config/settings.py), otherwise log files with be cleaned up at the start of the next test run.)
### <img src="https://seleniumbase.io/img/sb_icon.png" title="SeleniumBase" width="30" /> Demo Mode: ### <img src="https://seleniumbase.io/img/sb_icon.png" title="SeleniumBase" width="30" /> Demo Mode:
@ -214,14 +233,15 @@ pytest --reruns=2 --reruns-delay=1
### <img src="https://seleniumbase.io/img/sb_icon.png" title="SeleniumBase" width="30" /> Debugging tests: ### <img src="https://seleniumbase.io/img/sb_icon.png" title="SeleniumBase" width="30" /> Debugging tests:
**You can use the following code snippets in your scripts to help you debug issues:** You can use the following calls in your scripts to help you debug issues:
```python ```python
import time; time.sleep(5) # Makes the test wait and do nothing for 5 seconds. import time; time.sleep(5) # Makes the test wait and do nothing for 5 seconds.
import ipdb; ipdb.set_trace() # Enter debugging mode. n = next, c = continue, s = step. import ipdb; ipdb.set_trace() # Enter debugging mode. n = next, c = continue, s = step.
import pytest; pytest.set_trace() # Enter debugging mode. n = next, c = continue, s = step. import pytest; pytest.set_trace() # Enter debugging mode. n = next, c = continue, s = step.
``` ```
**To pause an active test that throws an exception or error, add ``--pdb -s``:** To pause an active test that throws an exception or error, add ``--pdb -s``:
```bash ```bash
pytest my_first_test.py --pdb -s pytest my_first_test.py --pdb -s

14
seleniumbase/plugins/base_plugin.py
View File

@ -13,14 +13,14 @@ from seleniumbase.fixtures import constants, errors
class Base(Plugin): class Base(Plugin):
""" """
This parser plugin includes the following command-line options for Nose: This parser plugin includes the following command-line options for Nose:
--env=ENV (Set a test environment. Use "self.env" to use this in tests.) --env=ENV (Set the test env. Access with "self.env" in tests.)
--data=DATA (Extra data to pass to tests. Use "self.data" in tests.) --data=DATA (Extra test data. Access with "self.data" in tests.)
--var1=DATA (Extra data to pass to tests. Use "self.var1" in tests.) --var1=DATA (Extra test data. Access with "self.var1" in tests.)
--var2=DATA (Extra data to pass to tests. Use "self.var2" in tests.) --var2=DATA (Extra test data. Access with "self.var2" in tests.)
--var3=DATA (Extra data to pass to tests. Use "self.var3" in tests.) --var3=DATA (Extra test data. Access with "self.var3" in tests.)
--settings-file=FILE (Overrides SeleniumBase settings.py values.) --settings-file=FILE (Override default SeleniumBase settings.)
--archive-logs (Archive old log files instead of deleting them.) --archive-logs (Archive old log files instead of deleting them.)
--report (The option to create a fancy report after tests complete.) --report (Create a fancy nosetests report after tests complete.)
--show-report If self.report is turned on, then the report will --show-report If self.report is turned on, then the report will
display immediately after tests complete their run. display immediately after tests complete their run.
Only use this when running tests locally, as this will Only use this when running tests locally, as this will

60
seleniumbase/plugins/pytest_plugin.py
View File

@ -11,50 +11,50 @@ from seleniumbase.fixtures import constants
def pytest_addoption(parser): def pytest_addoption(parser):
""" """
This parser plugin includes the following command-line options for pytest: This plugin adds the following command-line options to pytest:
--browser=BROWSER (The web browser to use.) --browser=BROWSER (The web browser to use. Default: "chrome")
--cap-file=FILE (The web browser's desired capabilities to use.) --cap-file=FILE (The web browser's desired capabilities to use.)
--cap-string=STRING (The web browser's desired capabilities to use.) --cap-string=STRING (The web browser's desired capabilities to use.)
--settings-file=FILE (Overrides SeleniumBase settings.py values.) --settings-file=FILE (Override default SeleniumBase settings.)
--env=ENV (Set a test environment. Use "self.env" to use this in tests.) --env=ENV (Set the test env. Access with "self.env" in tests.)
--data=DATA (Extra data to pass to tests. Use "self.data" in tests.) --data=DATA (Extra test data. Access with "self.data" in tests.)
--var1=DATA (Extra data to pass to tests. Use "self.var1" in tests.) --var1=DATA (Extra test data. Access with "self.var1" in tests.)
--var2=DATA (Extra data to pass to tests. Use "self.var2" in tests.) --var2=DATA (Extra test data. Access with "self.var2" in tests.)
--var3=DATA (Extra data to pass to tests. Use "self.var3" in tests.) --var3=DATA (Extra test data. Access with "self.var3" in tests.)
--user-data-dir=DIR (Set the Chrome user data directory to use.) --user-data-dir=DIR (Set the Chrome user data directory to use.)
--server=SERVER (The server / IP address used by the tests.) --server=SERVER (The server / IP address used by the tests.)
--port=PORT (The port that's used by the test server.) --port=PORT (The port that's used by the test server.)
--proxy=SERVER:PORT (This is the proxy server:port combo used by tests.) --proxy=SERVER:PORT (This is the proxy server:port combo used by tests.)
--agent=STRING (This designates the web browser's User Agent to use.) --agent=STRING (Modify the web browser's User-Agent string.)
--mobile (The option to use the mobile emulator while running tests.) --mobile (Use the mobile device emulator while running tests.)
--metrics=STRING ("CSSWidth,Height,PixelRatio" for mobile emulator tests.) --metrics=STRING (Set mobile "CSSWidth,CSSHeight,PixelRatio".)
--extension-zip=ZIP (Load a Chrome Extension .zip file, comma-separated.) --extension-zip=ZIP (Load a Chrome Extension .zip file, comma-separated.)
--extension-dir=DIR (Load a Chrome Extension directory, comma-separated.) --extension-dir=DIR (Load a Chrome Extension directory, comma-separated.)
--headless (The option to run tests headlessly. The default on Linux OS.) --headless (Run tests headlessly. Default mode on Linux OS.)
--headed (The option to run tests with a GUI on Linux OS.) --headed (Run tests with a GUI on Linux OS.)
--start-page=URL (The starting URL for the web browser when tests begin.) --start-page=URL (The starting URL for the web browser when tests begin.)
--archive-logs (Archive old log files instead of deleting them.) --archive-logs (Archive old log files instead of deleting them.)
--time-limit=SECONDS (Safely fail any test that exceeds the limit limit.) --time-limit=SECONDS (Safely fail any test that exceeds the limit limit.)
--slow (The option to slow down the automation.) --slow (Slow down the automation. Faster than using Demo Mode.)
--demo (The option to visually see test actions as they occur.) --demo (Slow down and visually see test actions as they occur.)
--demo-sleep=SECONDS (The option to wait longer after Demo Mode actions.) --demo-sleep=SECONDS (Set the wait time after Demo Mode actions.)
--highlights=NUM (Number of highlight animations for Demo Mode actions.) --highlights=NUM (Number of highlight animations for Demo Mode actions.)
--message-duration=SECONDS (The time length for Messenger alerts.) --message-duration=SECONDS (The time length for Messenger alerts.)
--check-js (The option to check for JavaScript errors after page loads.) --check-js (Check for JavaScript errors after page loads.)
--ad-block (The option to block some display ads after page loads.) --ad-block (Block some types of display ads after page loads.)
--block-images (The option to block images from loading during tests.) --block-images (Block images from loading during tests.)
--verify-delay=SECONDS (The delay before MasterQA verification checks.) --verify-delay=SECONDS (The delay before MasterQA verification checks.)
--disable-csp (This disables the Content Security Policy of websites.) --disable-csp (Disable the Content Security Policy of websites.)
--enable-sync (The option to enable "Chrome Sync".) --enable-sync (Enable "Chrome Sync".)
--use-auto-ext (The option to use Chrome's automation extension.) --use-auto-ext (Use Chrome's automation extension.)
--swiftshader (The option to use Chrome's "--use-gl=swiftshader" feature.) --swiftshader (Use Chrome's "--use-gl=swiftshader" feature.)
--incognito (The option to enable Chrome's Incognito mode.) --incognito (Enable Chrome's Incognito mode.)
--guest (The option to enable Chrome's Guest mode.) --guest (Enable Chrome's Guest mode.)
--devtools (The option to open Chrome's DevTools when the browser opens.) --devtools (Open Chrome's DevTools when the browser opens.)
--reuse-session (The option to reuse the browser session between tests.) --reuse-session / --rs (Reuse the browser session between tests.)
--crumbs (Option to delete all cookies between tests reusing a session.) --crumbs (Delete all cookies between tests reusing a session.)
--maximize (The option to start with the web browser maximized.) --maximize (Start tests with the web browser window maximized.)
--save-screenshot (The option to save a screenshot after each test.) --save-screenshot (Save a screenshot at the end of each test.)
--visual-baseline (Set the visual baseline for Visual/Layout tests.) --visual-baseline (Set the visual baseline for Visual/Layout tests.)
--timeout-multiplier=MULTIPLIER (Multiplies the default timeout values.) --timeout-multiplier=MULTIPLIER (Multiplies the default timeout values.)
""" """