SeleniumBase/help_docs/customizing_test_runs.md

548 lines
28 KiB
Markdown
Raw Permalink Normal View History

2022-11-05 13:27:22 +08:00
<!-- SeleniumBase Docs -->
2022-09-16 11:02:16 +08:00
## [<img src="https://seleniumbase.github.io/img/logo6.png" title="SeleniumBase" width="32">](https://github.com/seleniumbase/SeleniumBase/) pytest options for SeleniumBase
2019-10-27 14:25:53 +08:00
2022-04-23 10:57:53 +08:00
🎛️ SeleniumBase's [pytest plugin](https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/plugins/pytest_plugin.py) lets you customize test runs from the CLI (Command-Line Interface), which adds options for setting/enabling the browser type, Dashboard Mode, Demo Mode, Headless Mode, Mobile Mode, Multi-threading Mode, Recorder Mode, reuse-session mode, proxy config, user agent config, browser extensions, html-report mode, and more.
2017-10-27 04:43:33 +08:00
2022-04-23 10:57:53 +08:00
🎛️ Here are some examples of configuring tests, which can be run from the [examples/](https://github.com/seleniumbase/SeleniumBase/tree/master/examples) folder:
2017-10-25 06:49:00 +08:00
```bash
2020-04-19 01:14:03 +08:00
# Run a test in Chrome (default browser)
2020-11-18 08:22:12 +08:00
pytest my_first_test.py
2017-10-25 06:49:00 +08:00
2020-04-19 01:14:03 +08:00
# Run a test in Firefox
2023-08-01 03:08:55 +08:00
pytest test_swag_labs.py --firefox
2019-12-09 17:04:03 +08:00
2019-12-10 12:30:02 +08:00
# Run a test in Demo Mode (highlight assertions)
pytest test_demo_site.py --demo
2018-03-09 06:47:00 +08:00
2019-12-10 12:30:02 +08:00
# Run a test in Headless Mode (invisible browser)
2020-11-18 08:22:12 +08:00
pytest test_demo_site.py --headless
2019-12-09 17:04:03 +08:00
# Run tests multi-threaded using [n] threads
2023-05-25 17:23:29 +08:00
pytest test_suite.py -n4
2017-10-25 06:49:00 +08:00
2023-08-01 03:08:55 +08:00
# Reuse the browser session for all tests ("--reuse-session")
pytest test_suite.py --rs
2021-08-10 06:25:38 +08:00
2021-11-18 08:20:43 +08:00
# Reuse the browser session, but erase cookies between tests
2023-08-01 03:08:55 +08:00
pytest test_suite.py --rs --crumbs
2021-08-10 06:25:38 +08:00
# Create a real-time dashboard for test results
pytest test_suite.py --dashboard
2019-12-09 17:04:03 +08:00
# Create a pytest html report after tests are done
2018-12-11 15:01:57 +08:00
pytest test_suite.py --html=report.html
2018-03-09 06:47:00 +08:00
2021-10-01 13:26:03 +08:00
# Activate Debug Mode on failures ("c" to continue)
pytest test_fail.py --pdb -s
2019-12-09 17:04:03 +08:00
# Rerun failing tests more times
pytest test_suite.py --reruns=1
2018-03-09 06:47:00 +08:00
2021-10-01 13:26:03 +08:00
# Activate Debug Mode as the test begins ("n": next. "c": continue)
pytest test_null.py --trace -s
# Activate Recorder/Debug Mode as the test begins ("c" to continue)
pytest test_null.py --recorder --trace -s
# Pass extra data into tests (retrieve by calling self.data)
2019-12-09 17:04:03 +08:00
pytest my_first_test.py --data="ABC,DEF"
2018-10-11 01:22:02 +08:00
2019-12-09 17:04:03 +08:00
# Run tests on a local Selenium Grid
2020-02-18 15:10:36 +08:00
pytest test_suite.py --server="127.0.0.1"
2019-01-05 08:01:18 +08:00
2019-12-09 17:04:03 +08:00
# Run tests on a remote Selenium Grid
2018-12-11 15:01:57 +08:00
pytest test_suite.py --server=IP_ADDRESS --port=4444
2018-08-01 02:47:10 +08:00
2019-12-09 17:04:03 +08:00
# Run tests on a remote Selenium Grid with authentication
pytest test_suite.py --server=USERNAME:KEY@IP_ADDRESS --port=80
2019-12-09 17:04:03 +08:00
# Run tests through a proxy server
2019-02-28 17:52:50 +08:00
pytest proxy_test.py --proxy=IP_ADDRESS:PORT
2019-12-09 17:04:03 +08:00
# Run tests through a proxy server with authentication
2019-02-28 17:52:50 +08:00
pytest proxy_test.py --proxy=USERNAME:PASSWORD@IP_ADDRESS:PORT
2018-08-21 13:57:33 +08:00
2019-12-09 17:04:03 +08:00
# Run tests while setting the web browser's User Agent
pytest user_agent_test.py --agent="USER-AGENT-STRING"
2018-10-18 06:22:34 +08:00
2019-12-09 17:04:03 +08:00
# Run tests using Chrome's mobile device emulator (default settings)
pytest test_swag_labs.py --mobile
# Run mobile tests specifying CSS Width, CSS Height, and Pixel-Ratio
pytest test_swag_labs.py --mobile --metrics="360,640,2"
2019-12-09 17:04:03 +08:00
2023-07-19 03:19:23 +08:00
# Run a test with an option to evade bot-detection services
pytest verify_undetected.py --uc
2019-12-09 17:04:03 +08:00
# Run tests while changing SeleniumBase default settings
pytest my_first_test.py --settings-file=custom_settings.py
2017-10-25 06:49:00 +08:00
```
2022-04-23 10:57:53 +08:00
🎛️ You can interchange ``pytest`` with ``nosetests`` for most tests, but using ``pytest`` is recommended. (``chrome`` is the default browser if not specified.)
2017-10-25 06:49:00 +08:00
2022-04-23 10:57:53 +08:00
🎛️ If you're using ``pytest`` for running tests outside of the SeleniumBase repo, you'll want a copy of [pytest.ini](https://github.com/seleniumbase/SeleniumBase/blob/master/pytest.ini) at the base of the new folder structure. If using ``nosetests``, the same applies for [setup.cfg](https://github.com/seleniumbase/SeleniumBase/blob/master/setup.cfg).
2022-04-23 10:57:53 +08:00
🎛️ Here are some useful command-line options that come with ``pytest``:
2017-10-25 06:49:00 +08:00
2020-04-19 01:14:03 +08:00
```bash
2022-12-24 11:59:16 +08:00
-v # Verbose mode. Prints the full name of each test and shows more details.
-q # Quiet mode. Print fewer details in the console output when running tests.
2020-04-19 01:14:03 +08:00
-x # Stop running the tests after the first failure is reached.
2020-05-10 08:15:32 +08:00
--html=report.html # Creates a detailed pytest-html report after tests finish.
2023-08-01 03:08:55 +08:00
--co | --collect-only # Show what tests would get run. (Without running them)
--co -q # (Both options together!) - Do a dry run with full test names shown.
2020-04-19 01:14:03 +08:00
-n=NUM # Multithread the tests using that many threads. (Speed up test runs!)
2020-05-10 08:15:32 +08:00
-s # See print statements. (Should be on by default with pytest.ini present.)
--junit-xml=report.xml # Creates a junit-xml report after tests finish.
2022-06-27 00:16:57 +08:00
--pdb # If a test fails, enter Post Mortem Debug Mode. (Don't use with CI!)
--trace # Enter Debug Mode at the beginning of each test. (Don't use with CI!)
2020-08-09 01:30:48 +08:00
-m=MARKER # Run tests with the specified pytest marker.
2020-04-19 01:14:03 +08:00
```
2022-04-23 10:57:53 +08:00
🎛️ SeleniumBase provides additional ``pytest`` command-line options for tests:
2020-04-19 01:14:03 +08:00
```bash
2020-08-11 11:56:32 +08:00
--browser=BROWSER # (The web browser to use. Default: "chrome".)
2021-01-24 15:53:21 +08:00
--chrome # (Shortcut for "--browser=chrome". On by default.)
--edge # (Shortcut for "--browser=edge".)
--firefox # (Shortcut for "--browser=firefox".)
--safari # (Shortcut for "--browser=safari".)
--settings-file=FILE # (Override default SeleniumBase settings.)
2022-02-02 11:37:07 +08:00
--env=ENV # (Set the test env. Access with "self.env" in tests.)
--account=STR # (Set account. Access with "self.account" in tests.)
--data=STRING # (Extra test data. Access with "self.data" in tests.)
--var1=STRING # (Extra test data. Access with "self.var1" in tests.)
--var2=STRING # (Extra test data. Access with "self.var2" in tests.)
--var3=STRING # (Extra test data. Access with "self.var3" in tests.)
2022-05-14 06:05:12 +08:00
--variables=DICT # (Extra test data. Access with "self.variables".)
2020-04-19 01:14:03 +08:00
--user-data-dir=DIR # (Set the Chrome user data directory to use.)
--protocol=PROTOCOL # (The Selenium Grid protocol: http|https.)
2020-08-11 11:56:32 +08:00
--server=SERVER # (The Selenium Grid server/IP used for tests.)
--port=PORT # (The Selenium Grid port used by the test server.)
--cap-file=FILE # (The web browser's desired capabilities to use.)
--cap-string=STRING # (The web browser's desired capabilities to use.)
--proxy=SERVER:PORT # (Connect to a proxy server:port as tests are running)
--proxy=USERNAME:PASSWORD@SERVER:PORT # (Use an authenticated proxy server)
--proxy-bypass-list=STRING # (";"-separated hosts to bypass, Eg "*.foo.com")
2022-06-29 09:43:53 +08:00
--proxy-pac-url=URL # (Connect to a proxy server using a PAC_URL.pac file.)
--proxy-pac-url=USERNAME:PASSWORD@URL # (Authenticated proxy with PAC URL.)
--proxy-driver # (If a driver download is needed, will use: --proxy=PROXY.)
--multi-proxy # (Allow multiple authenticated proxies when multi-threaded.)
2021-03-24 03:14:13 +08:00
--agent=STRING # (Modify the web browser's User-Agent string.)
--mobile # (Use the mobile device emulator while running tests.)
2020-11-18 08:22:12 +08:00
--metrics=STRING # (Set mobile metrics: "CSSWidth,CSSHeight,PixelRatio".)
--chromium-arg="ARG=N,ARG2" # (Set Chromium args, ","-separated, no spaces.)
--firefox-arg="ARG=N,ARG2" # (Set Firefox args, comma-separated, no spaces.)
2021-05-23 02:15:36 +08:00
--firefox-pref=SET # (Set a Firefox preference:value set, comma-separated.)
2020-08-18 04:28:28 +08:00
--extension-zip=ZIP # (Load a Chrome Extension .zip|.crx, comma-separated.)
2020-04-19 01:14:03 +08:00
--extension-dir=DIR # (Load a Chrome Extension directory, comma-separated.)
--disable-features="F1,F2" # (Disable features, comma-separated, no spaces.)
--binary-location=PATH # (Set path of the Chromium browser binary to use.)
2023-08-06 03:39:04 +08:00
--driver-version=VER # (Set the chromedriver or uc_driver version to use.)
2022-10-02 14:08:15 +08:00
--sjw # (Skip JS Waits for readyState to be "complete" or Angular to load.)
2024-09-12 12:59:40 +08:00
--wfa # (Wait for AngularJS to be done loading after specific web actions.)
2022-08-27 10:02:41 +08:00
--pls=PLS # (Set pageLoadStrategy on Chrome: "normal", "eager", or "none".)
--headless # (Run tests in headless mode. The default arg on Linux OS.)
--headless2 # (Use the new headless mode, which supports extensions.)
--headed # (Run tests in headed/GUI mode on Linux OS, where not default.)
--xvfb # (Run tests using the Xvfb virtual display server on Linux OS.)
--locale=LOCALE_CODE # (Set the Language Locale Code for the web browser.)
2021-03-07 12:46:09 +08:00
--interval=SECONDS # (The autoplay interval for presentations & tour steps)
2020-04-19 01:14:03 +08:00
--start-page=URL # (The starting URL for the web browser when tests begin.)
2022-08-27 10:02:41 +08:00
--archive-logs # (Archive existing log files instead of deleting them.)
--archive-downloads # (Archive old downloads instead of deleting them.)
2020-10-12 21:46:06 +08:00
--time-limit=SECONDS # (Safely fail any test that exceeds the time limit.)
--slow # (Slow down the automation. Faster than using Demo Mode.)
--demo # (Slow down and visually see test actions as they occur.)
2022-07-23 21:20:03 +08:00
--demo-sleep=SECONDS # (Set the wait time after Slow & Demo Mode actions.)
2020-04-19 01:14:03 +08:00
--highlights=NUM # (Number of highlight animations for Demo Mode actions.)
--message-duration=SECONDS # (The time length for Messenger alerts.)
--check-js # (Check for JavaScript errors after page loads.)
--ad-block # (Block some types of display ads from loading.)
2023-11-15 10:13:17 +08:00
--host-resolver-rules=RULES # (Set host-resolver-rules, comma-separated.)
--block-images # (Block images from loading during tests.)
--do-not-track # (Indicate to websites that you don't want to be tracked.)
2020-04-19 01:14:03 +08:00
--verify-delay=SECONDS # (The delay before MasterQA verification checks.)
2024-05-13 12:26:48 +08:00
--ee | --esc-end # (Lets the user end the current test via the ESC key.)
2021-09-20 15:22:58 +08:00
--recorder # (Enables the Recorder for turning browser actions into code.)
2022-05-18 03:09:43 +08:00
--rec-behave # (Same as Recorder Mode, but also generates behave-gherkin.)
--rec-sleep # (If the Recorder is enabled, also records self.sleep calls.)
2022-06-21 06:04:19 +08:00
--rec-print # (If the Recorder is enabled, prints output after tests end.)
--disable-js # (Disable JavaScript on websites. Pages might break!)
2020-10-22 21:37:35 +08:00
--disable-csp # (Disable the Content Security Policy of websites.)
--disable-ws # (Disable Web Security on Chromium-based browsers.)
--enable-ws # (Enable Web Security on Chromium-based browsers.)
--enable-sync # (Enable "Chrome Sync" on websites.)
--uc | --undetected # (Use undetected-chromedriver to evade bot-detection.)
--uc-cdp-events # (Capture CDP events when running in "--undetected" mode.)
2023-10-29 14:04:19 +08:00
--log-cdp # ("goog:loggingPrefs", {"performance": "ALL", "browser": "ALL"})
2022-12-02 05:06:30 +08:00
--remote-debug # (Sync to Chrome Remote Debugger chrome://inspect/#devices)
2023-10-02 09:41:09 +08:00
--ftrace | --final-trace # (Debug Mode after each test. Don't use with CI!)
2020-12-21 05:47:30 +08:00
--dashboard # (Enable the SeleniumBase Dashboard. Saved at: dashboard.html)
--dash-title=STRING # (Set the title shown for the generated dashboard.)
2023-05-06 23:09:39 +08:00
--enable-3d-apis # (Enables WebGL and 3D APIs.)
2023-10-07 04:28:15 +08:00
--swiftshader # (Chrome "--use-gl=angle" / "--use-angle=swiftshader-webgl")
2022-08-27 10:02:41 +08:00
--incognito # (Enable Chrome's Incognito mode.)
--guest # (Enable Chrome's Guest mode.)
2023-08-01 03:08:55 +08:00
--dark # (Enable Chrome's Dark mode.)
--devtools # (Open Chrome's DevTools when the browser opens.)
2023-07-19 03:19:23 +08:00
--rs | --reuse-session # (Reuse browser session for all tests.)
--rcs | --reuse-class-session # (Reuse session for tests in class.)
--crumbs # (Delete all cookies between tests reusing a session.)
2022-08-24 09:14:00 +08:00
--disable-beforeunload # (Disable the "beforeunload" event on Chrome.)
2022-07-26 20:31:11 +08:00
--window-size=WIDTH,HEIGHT # (Set the browser's starting window size.)
--maximize # (Start tests with the browser window maximized.)
2022-02-26 07:09:38 +08:00
--screenshot # (Save a screenshot at the end of each test.)
2022-11-02 14:08:37 +08:00
--no-screenshot # (No screenshots saved unless tests directly ask it.)
2020-04-19 01:14:03 +08:00
--visual-baseline # (Set the visual baseline for Visual/Layout tests.)
--wire # (Use selenium-wire's webdriver for replacing selenium webdriver.)
--external-pdf # (Set Chromium "plugins.always_open_pdf_externally":True.)
2020-04-19 01:14:03 +08:00
--timeout-multiplier=MULTIPLIER # (Multiplies the default timeout values.)
2022-07-15 09:19:29 +08:00
--list-fail-page # (After each failing test, list the URL of the failure.)
2020-04-19 01:14:03 +08:00
```
2020-04-19 01:14:03 +08:00
(For more details, see the full list of command-line options **[here](https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/plugins/pytest_plugin.py)**.)
2022-04-23 10:57:53 +08:00
🎛️ You can also view a list of popular ``pytest`` options for SeleniumBase by typing:
```bash
seleniumbase options
```
Or the short form:
```bash
sbase options
```
2021-08-01 23:52:05 +08:00
--------
2020-04-19 01:14:03 +08:00
2022-09-16 11:02:16 +08:00
<h3><img src="https://seleniumbase.github.io/img/green_logo.png" title="SeleniumBase" width="32" /> Example tests using Logging:</h3>
2018-08-01 02:47:10 +08:00
2021-08-02 00:13:01 +08:00
To see logging abilities, you can run a test suite that includes tests that fail on purpose:
2017-10-25 06:49:00 +08:00
```bash
2021-08-02 00:13:01 +08:00
pytest test_suite.py
2017-10-25 06:49:00 +08:00
```
2020-12-23 00:56:21 +08:00
🔵 During test failures, logs and screenshots from the most recent test run will get saved to the ``latest_logs/`` folder. If ``--archive-logs`` is specified (or if ARCHIVE_EXISTING_LOGS is set to True in [settings.py](https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/config/settings.py)), test logs will also get archived to the ``archived_logs/`` folder. Otherwise, the log files will be cleaned out when the next test run begins (by default).
2017-10-25 06:49:00 +08:00
2022-09-16 11:02:16 +08:00
<h3><img src="https://seleniumbase.github.io/img/green_logo.png" title="SeleniumBase" width="32" /> Demo Mode:</h3>
2017-10-25 06:49:00 +08:00
2023-07-09 05:46:27 +08:00
🔵 If any test is moving too fast for your eyes to see what's going on, you can run it in **Demo Mode** by adding ``--demo`` on the command line, which pauses the browser briefly between actions, highlights page elements being acted on, and lets you know what test assertions are happening in real time:
2017-10-25 06:49:00 +08:00
```bash
pytest my_first_test.py --demo
2017-10-25 06:49:00 +08:00
```
2023-07-09 05:46:27 +08:00
🔵 You can override the default wait time by either updating [settings.py](https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/config/settings.py) or by using ``--demo-sleep=NUM`` when using Demo Mode. (NOTE: If you use ``--demo-sleep=NUM`` without using ``--demo``, nothing will happen.)
2017-10-25 06:49:00 +08:00
```bash
pytest my_first_test.py --demo --demo-sleep=1.2
2017-10-25 06:49:00 +08:00
```
2022-09-16 11:02:16 +08:00
<h3><img src="https://seleniumbase.github.io/img/green_logo.png" title="SeleniumBase" width="32" /> Passing additional data to tests:</h3>
2018-08-01 02:47:10 +08:00
If you want to pass additional data from the command line to your tests, you can use ``--data=STRING``. Now inside your tests, you can use ``self.data`` to access that.
2022-09-16 11:02:16 +08:00
<h3><img src="https://seleniumbase.github.io/img/green_logo.png" title="SeleniumBase" width="32" /> Running tests multithreaded:</h3>
2018-08-01 02:47:10 +08:00
2023-05-25 17:23:29 +08:00
To run ``pytest`` with multiple processes, add ``-n=NUM``, ``-n NUM``, or ``-nNUM`` on the command line, where ``NUM`` is the number of CPUs you want to use.
```bash
pytest -n=8
pytest -n 8
pytest -n8
```
2018-08-01 02:47:10 +08:00
2023-07-09 05:46:27 +08:00
<h3><img src="https://seleniumbase.github.io/img/green_logo.png" title="SeleniumBase" width="32" /> How to retry failing tests automatically:</h3>
2019-01-05 08:01:18 +08:00
2023-07-26 07:38:50 +08:00
<p>You can use <code translate="no">pytest --reruns=NUM</code> to retry failing tests that many times. Add <code translate="no">--reruns-delay=SECONDS</code> to wait that many seconds between retries. Example:</p>
2020-10-12 06:53:54 +08:00
2021-02-11 12:24:32 +08:00
```bash
pytest --reruns=1 --reruns-delay=1
2019-01-05 08:01:18 +08:00
```
2022-09-16 11:02:16 +08:00
<h3><img src="https://seleniumbase.github.io/img/green_logo.png" title="SeleniumBase" width="32" /> Debugging tests:</h3>
2018-08-01 02:47:10 +08:00
2023-07-09 05:46:27 +08:00
🔵 You can use the following calls in your scripts to help you debug issues:
2017-10-25 06:49:00 +08:00
```python
import time; time.sleep(5) # Makes the test wait and do nothing for 5 seconds.
2023-07-09 05:46:27 +08:00
import pdb; pdb.set_trace() # Debug Mode. n: next, c: continue, s: step, u: up, d: down.
import pytest; pytest.set_trace() # Debug Mode. n: next, c: continue, s: step, u: up, d: down.
2017-10-25 06:49:00 +08:00
```
2023-07-09 05:46:27 +08:00
🔵 To pause an active test that throws an exception or error, (*and keep the browser window open while **Debug Mode** begins in the console*), add **``--pdb``** as a ``pytest`` option:
2017-10-25 06:49:00 +08:00
```bash
2023-07-09 05:46:27 +08:00
pytest test_fail.py --pdb
2017-10-25 06:49:00 +08:00
```
2023-07-09 05:46:27 +08:00
🔵 To start tests in Debug Mode, add **``--trace``** as a ``pytest`` option:
```bash
pytest test_coffee_cart.py --trace
```
(**``pdb``** commands: ``n``, ``c``, ``s``, ``u``, ``d`` => ``next``, ``continue``, ``step``, ``up``, ``down``).
2017-10-25 06:49:00 +08:00
2022-09-16 11:02:16 +08:00
<h3><img src="https://seleniumbase.github.io/img/green_logo.png" title="SeleniumBase" width="32" /> Combinations of options:</h3>
2021-08-02 00:13:01 +08:00
2022-04-23 10:57:53 +08:00
🎛️ There are times when you'll want to combine various command-line options for added effect.
2023-05-25 17:23:29 +08:00
For instance, the multi-process option, ``-n8``, can be customized by adding:
2021-08-02 00:13:01 +08:00
``--dist=loadscope`` or ``--dist=loadfile`` to it.
2023-06-23 01:34:35 +08:00
There's more info on that here: [pytest-xdist](https://pypi.org/project/pytest-xdist/2.5.0/):
2021-08-02 00:13:01 +08:00
2023-05-25 17:23:29 +08:00
* ``-n8 --dist=loadscope``: Tests are grouped by module for test functions and by class for test methods. Groups are distributed to available workers as whole units. This guarantees that all tests in a group run in the same process. This can be useful if you have expensive module-level or class-level fixtures. Grouping by class takes priority over grouping by module.
2021-08-02 00:13:01 +08:00
2023-05-25 17:23:29 +08:00
* ``-n8 --dist=loadfile``: Tests are grouped by their containing file. Groups are distributed to available workers as whole units. This guarantees that all tests in a file run in the same worker.
2021-08-02 00:13:01 +08:00
2023-06-23 01:34:35 +08:00
<details>
2023-07-26 07:38:50 +08:00
<summary> ▶️ <code translate="no">-n8 --dist=loadgroup</code> (<b>click to expand</b>)</summary>
2023-06-23 01:34:35 +08:00
<div>
2023-07-26 07:38:50 +08:00
<ul><li>Tests are grouped by the <code translate="no">xdist_group</code> mark. Groups are distributed to available workers as whole units. This guarantees that all tests with the same <code translate="no">xdist_group</code> name run in the same worker.</li></ul>
2023-06-23 01:34:35 +08:00
```python
@pytest.mark.xdist_group(name="group1")
def test_1():
pass
class Test:
@pytest.mark.xdist_group("group1")
def test_2():
pass
```
2023-07-26 07:38:50 +08:00
<blockquote><p>This makes <code translate="no">test_1</code> and <code translate="no">Test::test_2</code> run in the same worker. Tests without the <code translate="no">xdist_group</code> mark are distributed normally.</p></blockquote>
2023-06-23 01:34:35 +08:00
</div>
</details>
2022-04-23 10:57:53 +08:00
🎛️ You might also want to combine multiple options at once. For example:
2021-08-02 00:13:01 +08:00
```bash
2023-05-25 17:23:29 +08:00
pytest --headless -n8 --dashboard --html=report.html -v --rs --crumbs
2021-08-02 00:13:01 +08:00
```
The above not only runs tests in parallel processes, but it also tells tests in the same process to share the same browser session, runs the tests in headless mode, displays the full name of each test on a separate line, creates a realtime dashboard of the test results, and creates a full report after all tests complete.
2020-12-18 14:27:39 +08:00
--------
2022-09-16 11:02:16 +08:00
<h3><img src="https://seleniumbase.github.io/img/green_logo.png" title="SeleniumBase" width="32" /> The SeleniumBase Dashboard:</h3>
2020-12-18 14:27:39 +08:00
2020-12-23 00:56:21 +08:00
🔵 The ``--dashboard`` option for pytest generates a SeleniumBase Dashboard located at ``dashboard.html``, which updates automatically as tests run and produce results. Example:
2020-12-19 13:14:42 +08:00
```bash
pytest --dashboard --rs --headless
```
2020-12-18 14:27:39 +08:00
2022-09-16 11:02:16 +08:00
<img src="https://seleniumbase.github.io/cdn/img/dashboard_1.png" alt="The SeleniumBase Dashboard" title="The SeleniumBase Dashboard" width="360" />
2020-12-18 14:27:39 +08:00
2020-12-23 00:56:21 +08:00
🔵 Additionally, you can host your own SeleniumBase Dashboard Server on a port of your choice. Here's an example of that using Python 3's ``http.server``:
2020-12-18 14:27:39 +08:00
```bash
python -m http.server 1948
```
2023-06-30 07:23:21 +08:00
🔵 Now you can navigate to ``http://localhost:1948/dashboard.html`` in order to view the dashboard as a web app. This requires two different terminal windows: one for running the server, and another for running the tests, which should be run from the same directory. (Use <kbd>Ctrl+C</kbd> to stop the http server.)
2020-12-18 14:27:39 +08:00
2020-12-23 00:56:21 +08:00
🔵 Here's a full example of what the SeleniumBase Dashboard may look like:
2020-12-18 14:27:39 +08:00
```bash
pytest test_suite.py --dashboard --rs --headless
```
2022-09-16 11:02:16 +08:00
<img src="https://seleniumbase.github.io/cdn/img/dashboard_2.png" alt="The SeleniumBase Dashboard" title="The SeleniumBase Dashboard" width="480" />
2020-12-18 14:27:39 +08:00
--------
2022-09-16 11:02:16 +08:00
<h3><img src="https://seleniumbase.github.io/img/green_logo.png" title="SeleniumBase" width="32" /> Pytest Reports:</h3>
2017-10-25 06:49:00 +08:00
2020-12-23 00:56:21 +08:00
🔵 Using ``--html=report.html`` gives you a fancy report of the name specified after your test suite completes.
2017-10-25 06:49:00 +08:00
```bash
2018-12-11 15:01:57 +08:00
pytest test_suite.py --html=report.html
2017-10-25 06:49:00 +08:00
```
2020-10-12 06:53:54 +08:00
2022-09-16 11:02:16 +08:00
<img src="https://seleniumbase.github.io/cdn/img/html_report.png" alt="Example Pytest Report" title="Example Pytest Report" width="520" />
2017-10-25 06:49:00 +08:00
2020-12-23 00:56:21 +08:00
🔵 When combining pytest html reports with SeleniumBase Dashboard usage, the pie chart from the Dashboard will get added to the html report. Additionally, if you set the html report URL to be the same as the Dashboard URL when also using the dashboard, (example: ``--dashboard --html=dashboard.html``), then the Dashboard will become an advanced html report when all the tests complete.
2020-12-19 13:14:42 +08:00
2020-12-23 00:56:21 +08:00
🔵 Here's an example of an upgraded html report:
2020-12-22 07:03:13 +08:00
```bash
pytest test_suite.py --dashboard --html=report.html
```
2022-09-16 11:02:16 +08:00
<img src="https://seleniumbase.github.io/cdn/img/dash_report.jpg" alt="Dashboard Pytest HTML Report" title="Dashboard Pytest HTML Report" width="520" />
2020-12-22 07:03:13 +08:00
2020-12-19 13:14:42 +08:00
If viewing pytest html reports in [Jenkins](https://www.jenkins.io/), you may need to [configure Jenkins settings](https://stackoverflow.com/a/46197356) for the html to render correctly. This is due to [Jenkins CSP changes](https://www.jenkins.io/doc/book/system-administration/security/configuring-content-security-policy/).
You can also use ``--junit-xml=report.xml`` to get an xml report instead. Jenkins can use this file to display better reporting for your tests.
```bash
pytest test_suite.py --junit-xml=report.xml
```
2020-12-18 14:27:39 +08:00
--------
2022-09-16 11:02:16 +08:00
<h3><img src="https://seleniumbase.github.io/img/green_logo.png" title="SeleniumBase" width="32" /> Nosetest Reports:</h3>
2017-10-25 06:49:00 +08:00
2018-03-05 06:55:00 +08:00
The ``--report`` option gives you a fancy report after your test suite completes.
2017-10-25 06:49:00 +08:00
```bash
2018-12-11 15:01:57 +08:00
nosetests test_suite.py --report
2017-10-25 06:49:00 +08:00
```
2020-10-12 06:53:54 +08:00
2022-09-16 11:02:16 +08:00
<img src="https://seleniumbase.github.io/cdn/img/nose_report.png" alt="Example Nosetest Report" title="Example Nosetest Report" width="320" />
2017-10-25 06:49:00 +08:00
2018-03-09 06:47:00 +08:00
(NOTE: You can add ``--show_report`` to immediately display Nosetest reports after the test suite completes. Only use ``--show_report`` when running tests locally because it pauses the test run.)
2020-12-18 14:27:39 +08:00
--------
2022-09-16 11:02:16 +08:00
<h3><img src="https://seleniumbase.github.io/img/green_logo.png" title="SeleniumBase" width="32" /> Language Locale Codes</h3>
2022-04-23 10:57:53 +08:00
You can specify a Language Locale Code to customize web pages on supported websites. With SeleniumBase, you can change the web browser's Locale on the command line by doing this:
```bash
pytest --locale=CODE # Example: --locale=ru
```
Visit <a href="https://github.com/seleniumbase/SeleniumBase/blob/master/help_docs/locale_codes.md"><b>🗾 Locales</b></a> for a full list of codes.
--------
2023-10-02 09:41:09 +08:00
<h3><img src="https://seleniumbase.github.io/img/green_logo.png" title="SeleniumBase" width="32" /> Changing the default driver version:</h3>
🔵 By default, SeleniumBase will make sure that the major driver version matches the major browser version for Chromium tests. (Eg. If Chrome `117.X` is installed and you have chromedriver `117.X`, then nothing happens, but if you had chromedriver `116.X` instead, then SeleniumBase would download chromedriver `117.X` to match the browser version.)
🎛️ To change this default behavior, you can use:
```bash
pytest --driver-version=VER
```
The `VER` in `--driver-version=VER` can be:
* A major driver version. Eg. `117`. (milestone)
* An exact driver version. Eg. `117.0.5938.92`.
* ``"browser"`` (exact match on browser version)
* ``"keep"`` (keep using the driver you already have)
* ``"latest"`` / ``"stable"`` (latest stable version)
* ``"previous"`` / ``"latest-1"`` (latest minus one)
* ``"beta"`` (latest beta version)
* ``"dev"`` (latest dev version)
* ``"canary"`` (latest canary version)
* ``"mlatest"`` (latest version for the milestone)
Note that different options could lead to the same result. (Eg. If you have the latest version of a browser for a milestone, then ``"browser"`` and ``"mlatest"`` should give you the same driver if the latest driver version for that milestone matches the browser version.)
--------
2022-09-16 11:02:16 +08:00
<h3><img src="https://seleniumbase.github.io/img/green_logo.png" title="SeleniumBase" width="32" /> Customizing default settings:</h3>
2021-08-01 23:52:05 +08:00
2022-04-23 10:57:53 +08:00
🎛️ An easy way to override [seleniumbase/config/settings.py](https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/config/settings.py) is by using a custom settings file.
2021-08-01 23:52:05 +08:00
Here's the command-line option to add to tests: (See [examples/custom_settings.py](https://github.com/seleniumbase/SeleniumBase/blob/master/examples/custom_settings.py))
2022-04-23 10:57:53 +08:00
```bash
pytest --settings-file=custom_settings.py
```
2021-08-01 23:52:05 +08:00
(Settings include default timeout values, a two-factor auth key, DB credentials, S3 credentials, and other important settings used by tests.)
--------
2022-09-16 11:02:16 +08:00
<h3><img src="https://seleniumbase.github.io/img/green_logo.png" title="SeleniumBase" width="32" /> Running tests on a remote Selenium Grid:</h3>
2021-08-01 23:52:05 +08:00
2023-10-29 13:41:32 +08:00
🌐 SeleniumBase lets you run tests on remote Selenium Grids such as [BrowserStack](https://www.browserstack.com/automate#)'s Selenium Grid, [Sauce Labs](https://saucelabs.com/products/platform-configurator)'s Selenium Grid, other Grids, and even your own Grid:
2021-08-01 23:52:05 +08:00
2022-04-23 10:57:53 +08:00
🌐 For setting browser desired capabilities while running Selenium remotely, see the ReadMe located here: https://github.com/seleniumbase/SeleniumBase/tree/master/examples/capabilities
2021-08-01 23:52:05 +08:00
Here's how to connect to a BrowserStack Selenium Grid server for running tests:
```bash
2021-09-07 00:58:41 +08:00
pytest test_demo_site.py --server=USERNAME:KEY@hub.browserstack.com --port=80
2021-08-01 23:52:05 +08:00
```
Here's how to connect to a Sauce Labs Selenium Grid server for running tests:
```bash
2021-09-07 00:58:41 +08:00
pytest test_demo_site.py --server=USERNAME:KEY@ondemand.us-east-1.saucelabs.com --port=443 --protocol=https
2021-08-01 23:52:05 +08:00
```
2021-11-18 08:20:43 +08:00
Here's how to connect to a CrossBrowserTesting Selenium Grid server for running tests:
```bash
pytest test_demo_site.py --server=USERNAME:KEY@hub.crossbrowsertesting.com --port=80
```
2022-04-23 10:57:53 +08:00
🌐 Or you can create your own Selenium Grid for test distribution. ([See this ReadMe for details](https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/utilities/selenium_grid/ReadMe.md))
2021-08-01 23:52:05 +08:00
2022-04-23 10:57:53 +08:00
🌐 To use a server on the ``https`` protocol, add ``--protocol=https``: (*Now automatic if the port is 443.*)
2021-08-01 23:52:05 +08:00
```bash
2021-09-07 00:58:41 +08:00
pytest test_demo_site.py --protocol=https --server=IP_ADDRESS --port=PORT
2021-08-01 23:52:05 +08:00
```
--------
2022-09-16 11:02:16 +08:00
<h3><img src="https://seleniumbase.github.io/img/green_logo.png" title="SeleniumBase" width="32" /> Using a Proxy Server:</h3>
2018-03-09 06:47:00 +08:00
2022-04-23 10:57:53 +08:00
🌐 If you wish to use a proxy server for your browser tests (Chromium or Firefox), you can add ``--proxy=IP_ADDRESS:PORT`` as an argument on the command line.
2018-03-09 06:47:00 +08:00
```bash
pytest proxy_test.py --proxy=IP_ADDRESS:PORT
```
2022-04-23 10:57:53 +08:00
🌐 If the proxy server that you wish to use requires authentication, you can do the following (Chromium only):
2018-12-14 08:08:56 +08:00
```bash
pytest proxy_test.py --proxy=USERNAME:PASSWORD@IP_ADDRESS:PORT
```
2022-04-23 10:57:53 +08:00
🌐 SeleniumBase also supports SOCKS4 and SOCKS5 proxies:
```bash
pytest proxy_test.py --proxy="socks4://IP_ADDRESS:PORT"
pytest proxy_test.py --proxy="socks5://IP_ADDRESS:PORT"
```
2018-03-09 06:47:00 +08:00
To make things easier, you can add your frequently-used proxies to PROXY_LIST in [proxy_list.py](https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/config/proxy_list.py), and then use ``--proxy=KEY_FROM_PROXY_LIST`` to use the IP_ADDRESS:PORT of that key.
```bash
pytest proxy_test.py --proxy=proxy1
```
2019-02-28 17:52:50 +08:00
2020-12-18 14:27:39 +08:00
--------
2022-09-16 11:02:16 +08:00
<h3><img src="https://seleniumbase.github.io/img/green_logo.png" title="SeleniumBase" width="32" /> Changing the User-Agent:</h3>
2019-02-28 17:52:50 +08:00
2022-04-23 10:57:53 +08:00
🔤 If you wish to change the User-Agent for your browser tests (Chrome and Firefox only), you can add ``--agent="USER-AGENT-STRING"`` as an argument on the command line.
2019-02-28 17:52:50 +08:00
```bash
pytest user_agent_test.py --agent="Mozilla/5.0 (Nintendo 3DS; U; ; en) Version/1.7412.EU"
```
2019-12-09 17:04:03 +08:00
2022-09-16 11:02:16 +08:00
<h3><img src="https://seleniumbase.github.io/img/green_logo.png" title="SeleniumBase" width="32" /> Mobile Device Testing:</h3>
2019-12-09 17:04:03 +08:00
2022-04-23 10:57:53 +08:00
📱 Use ``--mobile`` to quickly run your tests using Chrome's mobile device emulator with default values for device metrics (CSS Width, CSS Height, Pixel-Ratio) and a default value set for the user agent. To configure the mobile device metrics, use ``--metrics="CSS_Width,CSS_Height,Pixel_Ratio"`` to set those values. You'll also be able to set the user agent with ``--agent="USER-AGENT-STRING"`` (a default user agent will be used if not specified). To find real values for device metrics, [see this GitHub Gist](https://gist.github.com/sidferreira/3f5fad525e99b395d8bd882ee0fd9d00). For a list of available user agent strings, [check out this page](https://developers.whatismybrowser.com/useragents/explore/).
2019-12-09 17:04:03 +08:00
```bash
# Run tests using Chrome's mobile device emulator (default settings)
pytest test_swag_labs.py --mobile
# Run mobile tests specifying CSS Width, CSS Height, and Pixel-Ratio
pytest test_swag_labs.py --mobile --metrics="411,731,3"
# Run mobile tests specifying the user agent
pytest test_swag_labs.py --mobile --agent="Mozilla/5.0 (Linux; Android 9; Pixel 3 XL)"
```
2020-10-12 06:53:54 +08:00
2022-06-21 06:04:19 +08:00
--------
2022-09-16 11:02:16 +08:00
[<img src="https://seleniumbase.github.io/cdn/img/fancy_logo_14.png" title="SeleniumBase" width="290">](https://github.com/seleniumbase/SeleniumBase)