The nightly benchmarking runs haven't been running recently due to
failures to import Qiskit. This happened immediately after the import
poisoning. The problem appears to be because the `all` extra in the
metadata references the `qiskit` package by name, since this is how we
avoid repeating ourselves in the definition of the extra. However, as
we note in `pyproject.toml`, when called on a local wheel file, this
causes the resoution to install `qiskit` from PyPI, which means
overwriting the package with a different version. We don't actually
need everything installed for the benchmarks, so we can just set the
rule to install only what is needed.
The qiskit-experiments package was used in two places in qiskit's asv
benchmarks, the state tomography benchmark and the randomized
benchmarking circuit compilation benchmark. However, qiskit-experiments
is a downstream dependency of qiskit and trying to leverage it as
benchmark load generation is problematic because it introduces a
bidirectional dependency. It turns out these benchmarks don't really
need to use qiskit-experiments, the state tomography benchmarks aren't
really qiskit benchmarks directly. They include post-processing runtime
from qiskit-experiments and they're a much better benchmark for
qiskit-experiments rather than qiskit. The randomized benchmarking
circuit compilation benchmark is only using qiskit-experiments as a
randomized benchmarking circuit generator and we can replace it with
just manual circuit generation. This commit makes these changes and
removes qiskit-experiments from the asv configuration.
Previously, the `uninstall` component of the asv configuration was
attempting to uninstall the package `qiskit`. This _is_ the name of the
project, but it's not the name of the relevant Python package, and so
was not being uninstalled correctly. This meant that asv's managed
environments would retain the previous build, unless the updating the
commit changed the version number of the package.
This behaviour caused our tracking bot to be benchmarking the state of
the package only at the commit points that bump version numbers on
`main`. Typically, this means it was only ever benchmarking the
release-candidate versions of the package since we migrated from the
metapackage.
The configuration is rewritten because several components of it needed
to changes as it came other, and it was mostly autogenerated or
out-of-date anyway.
We may want to tweak how requirements are managed and installed in the
future.
Black is applied to the test files, and various tweaks needed to satisfy
Terra's lint requirements.
This commit updates the asv configuration to support two recent changes
in the terra repo. The first is updating the supported python version
list to reflect the current versions supported by terra. Python 3.6 is
no longer supported and python 3.10 is now supported. Additionally,
after Qiskit/qiskit-terra#7658 merged setuptools-rust and Rust are now
being used to build compiled extensions. While cython is still being
used, it's use will be removed soon with Qiskit/qiskit-terra#7702. This
commit updates the build configuration to build the rust extension and
then build a wheel from it instead of building the cython extension.
In Qiskit/qiskit-metapackage#1324 as part of the updates to the randomized benchmarking
benchmarks the version string in the benchmark class was bumped to
the latest ignis version to do 2 things, first indicate the benchmark
code had significantly changed to differentiate new results from old
ones and second to update the ignis version to the latest release in the
benchmarks. However on the second point the ignis version installed was
never actually updated. This commit fixes this oversight and updates the
ignis version in the asv config and also updates the state tomography
benchmark version to indicate that we're running with ignis 0.6.0 now
instead of 0.2.0 that we had been using previously.
The asv configuration file was a bit out of date and we actually haven't
been running benchmarks since last week because of this. The
qiskit-terra repository changed the default branch from master to main.
This means when we go to run benchmarks there are no new commits to run
because the asv configuration was setup to track the "master" branch.
This commit fixes this and also updates the python versions to include
python 3.9 which is supported and we should enable running benchmarks
with 3.9 too.
This commit updates the supported python versions in the asv.conf. Since
the terra 0.16.0 release (qiskit 0.23.0) hasn't support running with 3.5 so
that is removed. Also qiskit has supported python 3.8 since the combined
0.12.0 release. So 3.5 is removed from the list and 3.8 is added (which
is quite overdue).
In Qiskit/qiskit-terra#3861 a new requirement was added to terra without
updating the requirements list in the setup.py. This causes issues when
trying to run the benchmarks for commits after that PR and prior to
Qiskit/qiskit-terra#4045 merging because it can't actually install
terra. This commit adds pygments to the asv.conf.json to ensure we can
have continuous benchmarking.
Qiskit/qiskit-terra#3508 added an inadvertant hard coded import-time
dependency on having seaborn installed if matplotlib is present even
though we do not use the visualization functions in benchmarking. On
the benchmarking machine used to generate results at
https://qiskit.github.io/qiskit/ matplotlib is present on the system
which causes a ImportError traceback for all commits after
Qiskit/qiskit-terra#3508. A fix for this has been proposed in
Qiskit/qiskit-terra#3538 but to benchmark the commits between
Qiskit/qiskit-terra#3508 and when Qiskit/qiskit-terra#3538 merges we'll
need to have seaborn present in the asv built venv. This commit adds
seaborn to the asv install config to make sure we generate data for
those commits.
Related to Qiskit/qiskit-terra#3525
* Add benchmarks for mapping passes
This commit adds new benchmark methods for mapping passes. There are 2
benchmarks for each analysis pass that track the run time of the pass
and the memory consumption. Then for transformation passes a third
benchmark is added to track the depth after the pass is run. For swap
mappers a 4th benchmark also keeping track of how many swap gates are
in the circuit.
* Fix lint
The linter fails because two passes have not been released yet, this
commit disables this check for the file so we can run the benchmark for
these new passes.
* Add python-constraint to asv venv
The csp layout pass uses python-constraint which is an optional
dependency. This needs to be manually installed to make the csp layout
pass benchmarks work, this commit adds it to the list of dependencies
installed.
This commit bumps the ignis version used in the randomized benchmarking
and state tomography asv benchmarks. These benchmarks use a pinned
version of ignis for consistent results since the benchmarks track
terra development. Now that ignis 0.2.0 has been released we should use
it to ensure that the benchmarks keep working moving forward.
* first version of benchmark/randomize_benchmarking.py
* fix lint error
* fixed a misprint
* updated benchmark test for RB
* fixed RB benchmark test following comment
* fix lint error
* Add qiskit-ignis to benchmark env
* removed nq from the parameters
* Correct setup() parameter order
* Only return circuits for transpilation
* nseeds = [1]
* Fix various issues with the benchmark
This commit fixes a number of issues with the benchmark. First it
removes single entry parameters from the parameter list, there's no
reason to make them parameters if we don't use multiple values. It also
fixes a small bug in the return from the build_rb_circuit() function
where we were only returning one sequence of circuits instead of the
whole set. It also adjusts how we use random values by properly setting
a seed for both circuit building and tranpile. The last thing it does is
for consistency it pins the ignis version to a single version and sets
the benchmark version number to match that. This way if/when we need to
bump the ignis version we use in the benchmarks we'll have to bump the
benchmark's version too to match. This way we can make sure we're
using the same ignis version for all benchmark results.
* Reduce the number of length vectors used to 1
This commit removes one of the two length vectors we were using for
constructing the RB circuits. This is done in the interest of time since
the RB benchmarking can be quite slow so this reduces the number of
input parameters we iterate over which decreases the total run time of
RB benchmarks. If in the future we improve the performance so that the
impact is minimal to the full asv run we can add back additional length
vectors for benchmarking.
* Update copyright header to use consistent format
* Add asv benchmarks
This commit adds an asv benchmark suite to the tests include in this
repo. This will enable us to track the performance of the overall qiskit
project over time as we make commits.
* Remove H2 and LiH dirs
* Fix most, but not all, lint
* Remove commented unused code
* Add QV circuit benchmark and fix asv build and some lint
* Make lint pass
* Purity isn't in quantum_info in 0.7
* Add qft and random circuit benchmarks
* Fix issue caused by namespace package and aer sticking around
* Make depth an independent variable for qv benchmarks
* Simplify install and uninstall stage
The install stage was going off the rails trying failing to install the
built wheel in certain scenarios. When this happened the empty namespace
package from qiskit-aer was leftover and when benchmarks ran there was
nothing leftover. To avoid this and simplify the install (and by
extension uninstall) configuration to just rely on the default working
behavior this commit removes the dependence on the qiskit-aer package.
The aer package wasn't even used directly in the benchmarks since we're
not actually running simulations (for obvious time reasons). Instead Aer
was just being used as a backend for the transpiler. But, there is no
reason we have to use Aer for that, and BasicAer works just as well.
* Clear build artifacts before install
* Fix lint failures for new pylint
* Switch back to pip for wheel builds
* Fix another lint issue
* Fix hopefully the last lint issue
Jake Lishman