* Add infrastructure for gates, instruction, and operations in Rust
This commit adds a native representation of Gates, Instruction, and
Operations to rust's circuit module. At a high level this works by
either wrapping the Python object in a rust wrapper struct that tracks
metadata about the operations (name, num_qubits, etc) and then for other
details it calls back to Python to get dynamic details like the
definition, matrix, etc. For standard library gates like Swap, CX, H,
etc this replaces the on-circuit representation with a new rust enum
StandardGate. The enum representation is much more efficient and has a
minimal memory footprint (just the enum variant and then any parameters
or other mutable state stored in the circuit instruction). All the gate
properties such as the matrix, definiton, name, etc are statically
defined in rust code based on the enum variant (which represents the
gate).
The use of an enum to represent standard gates does mean a change in
what we store on a CircuitInstruction. To represent a standard gate
fully we need to store the mutable properties of the existing Gate class
on the circuit instruction as the gate by itself doesn't contain this
detail. That means, the parameters, label, unit, duration, and condition
are added to the rust side of circuit instrucion. However no Python side
access methods are added for these as they're internal only to the Rust
code. In Qiskit 2.0 to simplify this storage we'll be able to drop, unit,
duration, and condition from the api leaving only label and parameters.
But for right now we're tracking all of the fields.
To facilitate working with circuits and gates full from rust the
setting the `operation` attribute of a `CircuitInstruction` object now
transltates the python object to an internal rust representation.
For standard gates this translates it to the enum form described earlier,
and for other circuit operations 3 new Rust structs: PyGate,
PyInstruction, and PyOperation are used to wrap the underlying Python
object in a Rust api. These structs cache some commonly accessed static
properties of the operation, such as the name, number of qubits, etc.
However for dynamic pieces, such as the definition or matrix, callback
to python to get a rust representation for those.
Similarly whenever the `operation` attribute is accessed from Python
it converts it back to the normal Python object representation. For
standard gates this involves creating a new instance of a Python object
based on it's internal rust representation. For the wrapper structs a
reference to the wrapped PyObject is returned.
To manage the 4 variants of operation (`StandardGate`, `PyGate`,
`PyInstruction`, and `PyOperation`) a new Rust trait `Operation` is
created that defines a standard interface for getting the properties
of a given circuit operation. This common interface is implemented for
the 4 variants as well as the `OperationType` enum which wraps all 4
(and is used as the type for `CircuitInstruction.operation` in the
rust code.
As everything in the `QuantumCircuit` data model is quite coupled moving
the source of truth for the operations to exist in Rust means that more
of the underlying `QuantumCircuit`'s responsibility has to move to Rust
as well. Primarily this involves the `ParameterTable` which was an
internal class for tracking which instructions in the circuit have a
`ParameterExpression` parameter so that when we go to bind parameters we
can lookup which operations need to be updated with the bind value.
Since the representation of those instructions now lives in Rust and
Python only recieves a ephemeral copy of the instructions the
ParameterTable had to be reimplemented in Rust to track the
instructions. This new parameter table maps the Parameter's uuid (as a
u128) as a unique identifier for each parameter and maps this to a
positional index in the circuit data to the underlying instruction using
that parameter. This is a bit different from the Python parameter table
which was mapping a parameter object to the id of the operation object
using that parmaeter. This also leads to a difference in the binding
mechanics as the parameter assignment was done by reference in the old
model, but now we need to update the entire instruction more explicitly
in rust. Additionally, because the global phase of a circuit can be
parameterized the ownership of global phase is moved from Python into
Rust in this commit as well.
After this commit the only properties of a circuit that are not defined
in Rust for the source of truth are the bits (and vars) of the circuit,
and when creating circuits from rust this is what causes a Python
interaction to still be required.
This commit does not translate the full standard library of gates as
that would make the pull request huge, instead this adds the basic
infrastructure for having a more efficient standard gate representation
on circuits. There will be follow up pull requests to add the missing
gates and round out support in rust.
The goal of this pull request is primarily to add the infrastructure for
representing the full circuit model (and dag model in the future) in
rust. By itself this is not expected to improve runtime performance (if
anything it will probably hurt performance because of extra type
conversions) but it is intended to enable writing native circuit
manipulations in Rust, including transpiler passes without needing
involvement from Python. Longer term this should greatly improve the
runtime performance and reduce the memory overhead of Qiskit. But,
this is just an early step towards that goal, and is more about
unlocking the future capability. The next steps after this commit are
to finish migrating the standard gate library and also update the
`QuantumCircuit` methods to better leverage the more complete rust
representation (which should help offset the performance penalty
introduced by this).
Fixes: #12205
* Fix Python->Rust Param conversion
This commit adds a custom implementation of the FromPyObject trait for
the Param enum. Previously, the Param trait derived it's impl of the
trait, but this logic wasn't perfect. In cases whern a
ParameterExpression was effectively a constant (such as `0 * x`) the
trait's attempt to coerce to a float first would result in those
ParameterExpressions being dropped from the circuit at insertion time.
This was a change in behavior from before having gates in Rust as the
parameters would disappear from the circuit at insertion time instead of
at bind time. This commit fixes this by having a custom impl for
FromPyObject that first tries to figure out if the parameter is a
ParameterExpression (or a QuantumCircuit) by using a Python isinstance()
check, then tries to extract it as a float, and finally stores a
non-parameter object; which is a new variant in the Param enum. This
new variant also lets us simplify the logic around adding gates to the
parameter table as we're able to know ahead of time which gate
parameters are `ParameterExpression`s and which are other objects (and
don't need to be tracked in the parameter table.
Additionally this commit tweaks two tests, the first is
test.python.circuit.library.test_nlocal.TestNLocal.test_parameters_setter
which was adjusted in the previous commit to workaround the bug fixed
by this commit. The second is test.python.circuit.test_parameters which
was testing that a bound ParameterExpression with a value of 0 defaults
to an int which was a side effect of passing an int input to symengine
for the bind value and not part of the api and didn't need to be
checked. This assertion was removed from the test because the rust
representation is only storing f64 values for the numeric parameters
and it is never an int after binding from the Python perspective it
isn't any different to have float(0) and int(0) unless you explicit
isinstance check like the test previously was.
* Fix qasm3 exporter for std gates without stdgates.inc
This commit fixes the handling of standard gates in Qiskit when the user
specifies excluding the use of the stdgates.inc file from the exported
qasm. Previously the object id of the standard gates were used to
maintain a lookup table of the global definitions for all the standard
gates explicitly in the file. However, the rust refactor means that
every time the exporter accesses `circuit.data[x].operation` a new
instance is returned. This means that on subsequent lookups for the
definition the gate definitions are never found. To correct this issue
this commit adds to the lookup table a fallback of the gate name +
parameters to do the lookup for. This should be unique for any standard
gate and not interfere with the previous logic that's still in place and
functional for other custom gate definitions.
While this fixes the logic in the exporter the test is still failing
because the test is asserting the object ids are the same in the qasm3
file, which isn't the case anymore. The test will be updated in a
subsequent commit to validate the qasm3 file is correct without using
a hardcoded object id.
* Fix base scheduler analysis pass duration setting
When ALAPScheduleAnalysis and ASAPScheduleAnalysis were setting the
duration of a gate they were doing `node.op.duration = duration` this
wasn't always working because if `node.op` was a standard gate it
returned a new Python object created from the underlying rust
representation. This commit fixes the passes so that they modify the
duration and then explicit set the operation to update it's rust
representation.
* Fix python lint
* Fix last failing qasm3 test for std gates without stdgates.inc
While the logic for the qasm3 exporter was fixed
in commit a6e69ba4c9 to handle the edge
case of a user specifying that the qasm exporter does not use the
stdgates.inc include file in the output, but also has qiskit's standard
gates in their circuit being exported. The one unit test to provide
coverage for that scenario was not passing because when an id was used
for the gate definitions in the qasm3 file it was being referenced
against a temporary created by accessing a standard gate from the
circuit and the ids weren't the same so the reference string didn't
match what the exporter generated. This commit fixes this by changing
the test to not do an exact string comparison, but instead a line by
line comparison that either does exact equality check or a regex search
for the expected line and the ids are checked as being any 15 character
integer.
* Remove superfluous comment
* Cache imported classes with GILOnceCell
* Remove unused python variables
* Add missing file
* Update QuantumCircuit gate methods to bypass Python object
This commit updates the QuantumCircuit gate methods which add a given
gate to the circuit to bypass the python gate object creation and
directly insert a rust representation of the gate. This avoids a
conversion in the rust side of the code. While in practice this is just
the Python side object creation and a getattr for the rust code to
determine it's a standard gate that we're skipping. This may add up over
time if there are a lot of gates being created by the method.
To accomplish this the rust code handling the mapping of rust
StandardGate variants to the Python classes that represent those gates
needed to be updated as well. By bypassing the python object creation
we need a fallback to populate the gate class for when a user access the
operation object from Python. Previously this mapping was only being
populated at insertion time and if we never insert the python object
(for a circuit created only via the methods) then we need a way to find
what the gate class is. A static lookup table of import paths and class names
are added to `qiskit_circuit::imports` module to faciliate this and
helper functions are added to facilitate interacting with the class
objects that represent each gate.
* Deduplicate gate matrix definitions
* Fix lint
* Attempt to fix qasm3 test failure
* Add compile time option to cache py gate returns for rust std gates
This commit adds a new rust crate feature flag for the qiskit-circuits
and qiskit-pyext that enables caching the output from
CircuitInstruction.operation to python space. Previously, for memory
efficiency we were reconstructing the python object on demand for every
access. This was to avoid carrying around an extra pointer and keeping
the ephemeral python object around longer term if it's only needed once.
But right now nothing is directly using the rust representation yet and
everything is accessing via the python interface, so recreating gate
objects on the fly has a huge performance penalty. To avoid that this
adds caching by default as a temporary solution to avoid this until we
have more usage of the rust representation of gates.
There is an inherent tension between an optimal rust representation
and something that is performant for Python access and there isn't a
clear cut answer on which one is better to optimize for. A build time
feature lets the user pick, if what we settle on for the default doesn't
agree with their priorities or use case. Personally I'd like to see us
disable the caching longer term (hopefully before releasing this
functionality), but that's dependent on a sufficent level of usage from
rust superseding the current Python space usage in the core of Qiskit.
* Add num_nonlocal_gates implementation in rust
This commit adds a native rust implementation to rust for the
num_nonlocal_gates method on QuantumCircuit. Now that we have a rust
representation of gates it is potentially faster to do the count because
the iteration and filtering is done rust side.
* Performance tuning circuit construction
This commit fixes some performance issues with the addition of standard
gates to a circuit. To workaround potential reference cycles in Python
when calling rust we need to check the parameters of the operation. This
was causing our fast path for standard gates to access the `operation`
attribute to get the parameters. This causes the gate to be eagerly
constructed on the getter. However, the reference cycle case can only
happen in situations without a standard gate, and the fast path for
adding standard gates directly won't need to run this so a skip is added
if we're adding a standard gate.
* Add back validation of parameters on gate methods
In the previous commit a side effect of the accidental eager operation
creation was that the parameter input for gates were being validated by
that. By fixing that in the previous commit the validation of input
parameters on the circuit methods was broken. This commit fixes that
oversight and adds back the validation.
* Skip validation on gate creation from rust
* Offload operation copying to rust
This commit fixes a performance regression in the
`QuantumCircuit.copy()` method which was previously using Python to copy
the operations which had extra overhead to go from rust to python and
vice versa. This moves that logic to exist in rust and improve the copy
performance.
* Fix lint
* Perform deepcopy in rust
This commit moves the deepcopy handling to occur solely in Rust.
Previously each instruction would be directly deepcopied by iterating
over the circuit data. However, we can do this rust side now and doing
this is more efficient because while we need to rely on Python to run a
deepcopy we can skip it for the Rust standard gates and rely on Rust to
copy those gates.
* Fix QuantumCircuit.compose() performance regression
This commit fixes a performance regression in the compose() method. This
was caused by the checking for classical conditions in the method
requiring eagerly converting all standard gates to a Python object. This
changes the logic to do this only if we know we have a condition (which
we can determine Python side now).
* Fix map_ops test case with no caching case
* Fix typos in docs
This commit fixes several docs typos that were caught during code review.
Co-authored-by: Eli Arbel <46826214+eliarbel@users.noreply.github.com>
* Shrink memory usage for extra mutable instruction state
This commit changes how we store the extra mutable instruction state
(condition, duration, unit, and label) for each `CircuitInstruction`
and `PackedInstruction` in the circuit. Previously it was all stored
as separate `Option<T>` fields on the struct, which required at least
a pointer's width for each field which was wasted space the majority of
the time as using these fields are not common. To optimize the memory
layout of the struct this moves these attributes to a new struct which
is put in an `Option<Box<_>>` which reduces it from 4 pointer widths
down to 1 per object. This comes from extra runtime cost from the extra
layer of pointer indirection but as this is the uncommon path this
tradeoff is fine.
* Remove Option<> from params field in CircuitInstruction
This commit removes the Option<> from the params field in
CircuitInstruction. There is no real distinction between an empty vec
and None in this case, so the option just added another layer in the API
that we didn't need to deal with. Also depending on the memory alignment
using an Option<T> might have ended up in a little extra memory usage
too, so removing it removes that potential source of overhead.
* Eagerly construct rust python wrappers in .append()
This commit updates the Python code in QuantumCircuit.append() method
to eagerly construct the rust wrapper objects for python defined circuit
operations.
* Simplify code around handling python errors in rust
* Revert "Skip validation on gate creation from rust"
This reverts commit 2f81bde8bf. The
validation skipping was unsound in some cases and could lead to invalid
circuit being generated. If we end up needing this as an optimization we
can remove this in the future in a follow-up PR that explores this in
isolation.
* Temporarily use git for qasm3 import
In Qiskit/qiskit-qasm3-import#34 the issue we're hitting caused by
qiskit-qasm3-import using the private circuit attributes removed in this
PR was fixed. This commit temporarily moves to installing it from git so
we can fully run CI. When qiskit-qasm3-import is released we should
revert this commit.
* Fix lint
* Fix lint for real (we really need to use a py312 compatible version of pylint)
* Fix test failure caused by incorrect lint fix
* Relax trait-method typing requirements
* Encapsulate `GILOnceCell` initialisers to local logic
* Simplify Interface for building circuit of standard gates in rust
* Simplify complex64 creation in gate_matrix.rs
This just switches Complex64::new(re, im) to be c64(re, im) to reduce
the amount of typing. c64 needs to be defined inplace so it can be a
const fn.
* Simplify initialization of array of elements that are not Copy (#28)
* Simplify initialization of array of elements that are not Copy
* Only generate array when necessary
* Fix doc typos
Co-authored-by: Kevin Hartman <kevin@hart.mn>
* Add conversion trait for OperationType -> OperationInput and simplify CircuitInstruction::replace()
* Use destructuring for operation_type_to_py extra attr handling
* Simplify trait bounds for map_indices()
The map_indices() method previously specified both Iterator and
ExactSizeIterator for it's trait bounds, but Iterator is a supertrait of
ExactSizeIterator and we don't need to explicitly list both. This commit
removes the duplicate trait bound.
* Make Qubit and Clbit newtype member public
As we start to use Qubit and Clbit for creating circuits from accelerate
and other crates in the Qiskit workspace we need to be able to create
instances of them. However, the newtype member BitType was not public
which prevented creating new Qubits. This commit fixes this by making it
public.
* Use snakecase for gate matrix names
* Remove pointless underscore prefix
* Use downcast instead of bound
* Rwork _append reference cycle handling
This commit reworks the multiple borrow handling in the _append() method
to leveraging `Bound.try_borrow()` to return a consistent error message
if we're unable to borrow a CircuitInstruction in the rust code meaning
there is a cyclical reference in the code. Previously we tried to detect
this cycle up-front which added significant overhead for a corner case.
* Make CircuitData.global_phase_param_index a class attr
* Use &[Param] instead of &SmallVec<..> for operation_type_and_data_to_py
* Have get_params_unsorted return a set
* Use lookup table for static property methods of StandardGate
* Use PyTuple::empty_bound()
* Fix lint
* Add missing test method docstring
* Reuse allocations in parameter table update
* Remove unnecessary global phase zeroing
* Move manually set params to a separate function
* Fix release note typo
* Use constant for global-phase index
* Switch requirement to release version
---------
Co-authored-by: Eli Arbel <46826214+eliarbel@users.noreply.github.com>
Co-authored-by: Jake Lishman <jake.lishman@ibm.com>
Co-authored-by: John Lapeyre <jlapeyre@users.noreply.github.com>
Co-authored-by: Kevin Hartman <kevin@hart.mn>
In the past couple of days we've seen the miri tests fail in CI while
attempting to build crossbeam-epoch from source. We need to do this to
ensure that crossbeam-epoch is miri safe so that we can run our own
tests of Qiskit's unsafe code in ci. This failure was likely an issue in
the recent nightly builds so this commit pins the nightly rust version
to one from last week when everything was known to be working. We can
remove this specific pin when we know that upstream rust has fixed the
issue.
This commit adds the missing config to the github actions workflow for
running required tests (currently only arm64 macOS test jobs) to the
merge queue. This is necessary to make the job required in the branch
protection rules, because the jobs will need to pass as part of the
merge queue too.
* Promote arm64 macOS to tier 1
Github recently added a new macOS runner that is using the m1 CPU that
is usable for open source projects. [1] Previously Qiskit had support
for arm64 macOS at tier 3 because we were only able to cross compile
for the platform and not test the binaries. Now that we can run CI jobs
on the platform we're able to run both unit tests and test our binaries
on release. This commit adds a new set of test jobs and wheel builds
that use the macos-14 runner that mirrors the existing x86_64 macOS
jobs we have. This brings arm64 macOS to the same support level as
x86_64. THe only difference here is that azure pipelines doesn't have
arm64 macOS support so the test job will run in github actions instead.
[1] https://github.blog/changelog/2024-01-30-github-actions-introducing-the-new-m1-macos-runner-available-to-open-source/
* Update test job
* Fix syntax error
* Use a string for python version in test job
* Disable fail-fast on test job
* DNM: Test wheel builds
* Skip universal builds
* Force system python to arm64 on macOS arm64 bwheel builds
* Correctly skip universal builds
* Skip python 3.8 arm64
* Add back cross build job just for python 3.8
* Add numpy env details to arm64 test job
* Use MSRV for one of the test jobs
* Fix build_pgo.sh script when running on arm64 mac
* Revert "DNM: Test wheel builds"
This reverts commit 97eaa6fee8.
* Rename macos arm py38 cross-build job
* Add Rust-based `SparsePauliOp.to_matrix`
This rewrites the numerical version of `SparsePauliOp.to_matrix` to be
written in parallelised Rust, building up the matrices row-by-row rather
than converting each contained operator to a matrix individually and
summing them.
The new algorithms are complete row-based, which is embarrassingly
parallel for dense matrices, and parallelisable with additional copies
and cumulative sums in the CSR case.
The two new algorithms are an asymptotic complexity improvement for both
dense and sparse matrices over the "sum the individual matrices"
version. In the dense case, the cost goes from
O(4 ** num_qubits * num_ops)
to
O(4 ** num_qubits + (2 ** num_qubits) * reduced_num_ops)
where the first term is from the zeroing of the matrix, and the second
is from updating the elements in-place. `reduced_num_ops` is the number
of explicitly stored operations after Pauli-operator uniqueness
compaction, so is upper-bounded as `4 ** num_qubits`. (The Pauli
compaction goes as `O(num_ops)`, so is irrelevant to the complexity
discussion.) The CSR form of the algorithm goes as
O(2 ** num_qubits * reduced_num_ops * lg(reduced_num_ops))
which (I think! - I didn't fully calculate it) is asymptotically the
same as before, but in practice the constant factors and intermediate
memory use are *dramatically* reduced, and the new algorithm is
threadable with an additional `O(2 ** num_qubits * reduced_num_ops)`
intermediate memory overhead (the serial form has only
`O(reduced_num_ops)` memory overhead).
The `object`-coefficients form is left as-is to avoid exploding the
complexity in Rust space; these objects are already slow and unsuited
for high-performance code, so the effects should be minimal.
* Add non-blocking Miri to CI
As we begin to include more `unsafe` code in the Rust-accelerated
components, it is becoming more important for us to test these in an
undefined-behaviour sanitiser. This is done in a separate CI job
because:
- we do not yet know how stable Miri will be, so we don't want to block
on it.
- some dependencies need their version-resolution patching to
Miri-compatible versions, but we want to run our regular test suites
with the same versions of packages we will be building against.
* Parallelise cumulative nnz sum
This parallelises the previously serial-only cumulative sum of the
`indptr` array of number of non-zero entries at the end. In practice, I
didn't actually see any change in performance from this, but
philosophically it feels like the right thing to do.
* Update Miri pin to later version of crossbeam-epohc
* Improve error handling and messages
* Simplify unnecessary match
* Add link to environment variable configuration
* Add link to Rayon plumbing README
* Add explicit test of serial and parallel modes
* switched from sphinx.ext.viewcode to sphinx.ext.linkcode
* removed extra line
* Add section header for source code links
Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>
* removed docstring
Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>
* update return string
Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>
* added back blank line
* Added a method to determine the GitHub branch
Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>
* add blank line
Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>
* remove print statement
Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>
* Try to fix error for contextlib file
We got this stacktrace:
Traceback (most recent call last):
File "/home/vsts/work/1/s/.tox/docs/lib/python3.8/site-packages/sphinx/events.py", line 96, in emit
results.append(listener.handler(self.app, *args))
File "/home/vsts/work/1/s/.tox/docs/lib/python3.8/site-packages/sphinx/ext/linkcode.py", line 55, in doctree_read
uri = resolve_target(domain, info)
File "/home/vsts/work/1/s/docs/conf.py", line 216, in linkcode_resolve
file_name = PurePath(full_file_name).relative_to(repo_root)
File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/pathlib.py", line 908, in relative_to
raise ValueError("{!r} does not start with {!r}"
ValueError: '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/contextlib.py' does not start with '/home/vsts/work/1/s'
We shouldn't even attempt to generate a link for the file contextlib.py
* Try to fix error for Jenkins run #20240221.52
New build failed with this stacktrace:
Traceback (most recent call last):
File "/home/vsts/work/1/s/.tox/docs/lib/python3.8/site-packages/sphinx/events.py", line 96, in emit
results.append(listener.handler(self.app, *args))
File "/home/vsts/work/1/s/.tox/docs/lib/python3.8/site-packages/sphinx/ext/linkcode.py", line 55, in doctree_read
uri = resolve_target(domain, info)
File "/home/vsts/work/1/s/docs/conf.py", line 215, in linkcode_resolve
full_file_name = inspect.getsourcefile(obj)
File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/inspect.py", line 696, in getsourcefile
filename = getfile(object)
File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/inspect.py", line 665, in getfile
raise TypeError('{!r} is a built-in class'.format(object))
TypeError: <class 'builtins.CustomClassical'> is a built-in class
So I added a condition that the obj should not be a built-in
* Check that qiskit in module name sooner
* moved valid code object verification earlier
* added try except statement to getattr call
* added extra try/except block
* Also support Azure Pipelines
* removed unused import
* Revert Azure support to keep things simple
* added extra "/" to final URL
* Move GitHub branch logic to GitHub Action
* switched to importlib and removed redundant block of code
* Apply suggestions from code review
Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>
* added back spaces
* Clarify docs_deploy GitHub logic
1. Remove misleading PR conditional. This worfklow doesn't even run in PRs. It was bad copy-pasta from the runtime repo
2. Clarify why we point tag builds to their stable branch name
* Use pathlib for relativizing file name
* Fix relative_to() path
* Remove tox prefix
---------
Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>
With Hamamura-san having moved on from IBM now, this commit removes him
from being tagged/mentioned as a primary contact/reviewer for the
modules he'd previously been working in. We're of course happy to keep
working together, just there's no expectation anymore!
This bumps `upload-artifact` and `download-artifact` to version 4. The
jump from version 3 to version 4 is a major incompatible change,
especially around uploading to an existing artefact, which we do during
our wheel builds. The new recommended pattern is to use different file
names for each call to `upload-artifact`, and match the download with a
glob match in `download-artifact`.
The pip uninstall command used in the qiskit neko job was missing the -y
flag used to force pip to run the uninstall. Instead it was failing on
the prompt to confirm the uninstall and then not uninstalling an
existing install because of the failure. This has been causing us to not
actually test the propsed PR because it was mixing the state of qiskit <
1.0 from pypi and >=1.0. Now that 1.0 has been released we're hitting
failures in CI because all the breaking API changes we introduced in 1.0
are finally being run and the test suite is and downstream packages we
were integration testing with are now broken. This commit fixes the
uninstall command (although arguably it probably could be removed). The
neko job will still fail with this applied as we'll need to fix it
upstream but this is currently blocked on a qiskit-nature release.
* Add script to report numpy env
* Rename to more descriptive name
* Report Numpy runtime status in CI
* Add threadpoolctl for BLAS information
---------
Co-authored-by: Jake Lishman <jake.lishman@ibm.com>
This are internal compile-time only dependencies, and in almost all
cases do not require a release note. This commit makes dependabot
automatically label its PRs as such.
* Set version number to 1.0.0rc1 for first release candidate
For the 1.0.0 release we're going to push release candidates prior to
the final to enable testing before we cut the final release. In
preparation for tagging the first release candidate this commit updates
the version string to indicate it's a release candidate. This commit
should be the final commit on main for 1.0.0rc1 and what gets tagged as
1.0.0rc1 post-merge.
* Bump cibuildwheel version
Recently github actions bumped their Windows CI images which includes a
new release of powershell. This version of powershell is incompatible
with the version of cibuildwheel we were previously using. 2.16.5 fixes
compatibility with the new powershell version, so bumping the version
used is a requirement for releasing now.
* Build release wheels for tier 1 platforms with PGO
This commit adds the infrastructure to build the wheels we publish to
PyPI at release time with PGO. Profile guided optimization (PGO) is a
compiler technique that uses collected data about a typical execution
to inform optimizations the compiler can make about building the output
binary. For more details on PGO you can refer to the Rust [1] and Clang
[2] documention on the topic.
To start the only data collected is from running the qiskit
unittests. This should give us some broad coverage as we'll exercise
everything as we do in the unittests. This also means as we grow our
Rust usage in Qiskit we'll continue to cover that code in the unittests
and ensure we have coverage in PGO too. However, longer term we'll
potentially want to look at dedicated PGO scripts that exercise Qiskit
at larger scale than what we can do in the unittests.
[1] https://doc.rust-lang.org/rustc/profile-guided-optimization.html
[2] https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization
* Add comment about per job overrides
* Make merged path configurable
* Update pyproject.toml
* Update tools/build_pgo.sh
* Add other primitives team members to CODEOWNERS
* Switch to using a team
Co-authored-by: Matthew Treinish <mtreinish@kortar.org>
---------
Co-authored-by: Matthew Treinish <mtreinish@kortar.org>
These directories in `CODEOWNERS` were initially migrated from Aqua, but
have since been removed from the repository (or repurposed, in the case
of `utils`). This removes the now-obsolete specific overrides from the
owners file, since they were no longer having a meaningful effect.
The randomised test suite does not to install the long-deprecated
`qiskit_ibmq_provider`, and doing so causes an old, incompatible version
of `qiskit-terra` to be installed, breaking the editable install.
* Replace `qiskit` metapackage with `qiskit-terra`
This commit completely removes the concept of the `qiskit-terra` package
from Qiskit main. The hitherto metapackage `qiskit` is promoted to be
the concrete code package.
This is a completely breaking change for packaging purposes for users,
as there is no clean upgrade path from `qiskit-terra` to `qiskit`; PyPI
and pip do not give us the tools to mark that the former obsoletes and
supersedes the latter. We intend to follow this PR with a technical
blog post somewhere explaining the changes, how users should adapt
("to install Qiskit 1.0, you must start a new venv"), and why we needed
to do this.
The "why", in part, is:
- the metapackage legacy causes awkward upgrade paths on every release;
some packages still depend on `qiskit-terra`, some on `qiskit`, and
pip will happily make those two get out-of-sync when upgrading a
transitive dependency with only a warning that users are used to
ignoring.
- with the 1.0 release, we (believe we) will have some leeway from users
to make such a breaking change.
- having the metapackage split makes it difficult for downstream
projects and developers to test against `main` - they always must
install both `qiskit-terra` and `qiskit`, and the latter has no
meaning in editable installs, so needs re-installing after each
version bump. Problems surrounding this have already broken CI for
Qiskit, for at least one internal IBM repo, and at least my dev
install of Qiskit. This could be improved a bit with more education,
but it's still always going to increase the barrier to entry and make
it much harder to do the right thing.
We will not publish any 1.0 or above version of `qiskit-terra`. All
dependents on Qiskit should switch their requirements to `qiskit`.
* Add manual uninstall for Neko
* Fix Windows paths
Co-authored-by: Matthew Treinish <mtreinish@kortar.org>
* Refer to stdlib documentation for odd shells
---------
Co-authored-by: Matthew Treinish <mtreinish@kortar.org>
Since we are using `build`, a basic PEP 517 builder, we do not need to
manually install the build requirements; our `pyproject.toml` specifies
them, and they'll be automatically pulled in.
With the pivot to PyPI trusted publishers in gh-10999, we no longer use
`twine` manually either.
* Add support for Python 3.12
Python 3.12.0 was released on 10-02-2023, this commit marks the start of
support for Python 3.12 in Qiskit. It adds the supported Pythonv ersion
in the package metadata and updates the CI configuration to run test
jobs on Python 3.12 and build Python 3.12 wheels on release.
Fixes: #10887
* Add release note
* Avoid deprecated `datetime.datetime.utcnow()` usage
In Python 3.12 `datetime.datetime.utcnow()` has been deprecated, being
replaced by: `datetime.datetime.now(datetime.UTC)`. This commit updates
the usage of `utcnow()` to follow the new convention.
* Adjust UTC usage to support Python 3.8
The recommended alternative for using utcnow() in the deprecation
warnings emitted by Python 3.12 are not compatible with Python 3.8. The
datetime.UTC alias was not added to Python until Python 3.11. To ensure
that the code is compatible with Python < 3.11 this commit updates all
the usage of datetime.UTC to use datetime.timezone.utc instead, which is
what datetime.UTC aliases to in Python >=3.11.
* Ensure metapackage is installed during CI and tox
This ensures that the local version of the metapackage is also built and
installed on all CI runs (and in `tox`, where it's overridden) so that
dependencies on the metapackage in our optionals (e.g. Aer) will not
cause the older released version of Terra to be installed.
`tox` does not like having two local packages under test simultaneously
through its default configuration, so this fakes things out by putting
the two packages in the run dependencies and setting `skip_install`.
* Fix sdist build
* Use regular installs for metapackage
* Simplify build requirements install
This sets all file access in the translatable strings bash script to be
via absolute paths, rather than relative paths that have proven
error-prone to update as the repository structure changes, especially as
multiple base directories and git repositories are used.
* Pivot to gha for wheel builds and pypi trusted publishers
This commit pivots the wheel publishing jobs to PyPI trusted publishers.
This mechanism authorizes the repository action directly so that user
creds or tokens are not needed to push the wheels anymore. This
mechanism will be more robust as the the github repository is linked
directly to the pypi project and not dependent on a single user account.
The one tradeoff required for this is that we must use github actions to
leverage this PyPI feature. So this commit migrates the wheel publishing
jobs from azure pipelines to github actions (which we were already using
for non-x86 linux wheels).
* Fix macOS image used in macOS jobs
* Fix nits in metapackage job
* Combine tier 1 platform builds into single upload job
This adds a GitHub Action that triggers whenever Mergify opens a PR
against one on the stable branches, and copies across the relevant
labels and milestone from the base PR (if any). This just ensures that
things like the changelog tag and the tracking milestone are reliably
tracked without a maintainer needing to do it manually.
* Fix issues with translation deployment script
During the 0.25.1/0.44.1 release the translation publishing step failed.
This was because of a parsing error with the ssh private key during the
decryption step. This seems to be due to how the action was configured
to load the private key post decryption into an env variable that github
actions would then use for it's checkout action to handle cloning with
authentication via ssh. This commit switches back to just using a
standalone bash script that has proven to be reliable historically and
also gives us tighter control on how commands get executed. As part of
this a new encrypted private key is added to the repo as the previous
key has been revoked and invalidated since the job failure during the
release.
* Apply suggestions from code review
Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>
Co-authored-by: Jake Lishman <jake@binhbar.com>
* Update tools/deploy_translatable_strings.sh
Co-authored-by: Jake Lishman <jake@binhbar.com>
---------
Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>
Co-authored-by: Jake Lishman <jake@binhbar.com>
This commit fixes an issue in the job definition for the new docs
deployment job added in #10610. The new jobs are conditioned to only
execute from the Qiskit/qiskit-terra repo. However that line is being
flagged as a syntax error in github. This seems to be due to the double
quotes used for `"Qiskit"`. A similar condition exists on the
pre-existing github actions jobs but they use a single quote instead of
a double and they function correctly. This commit updates the condition
to used single quotes to match the working syntax in other jobs.
* Add GitHub Actions documentation-deployment pipeline
This brings a documentation-deployment pipeline into the Qiskit/Terra
repository, allowing it to fully deploy its documentation to
`qiskit.org`, a task previously only the metapackage could perform.
This does not fully unify the documentation with files from the
metapackage, it just adds a pipeline to do the final deployment.
This includes a revitalised translatable-strings pipeline, which was
previously broken on the metapackage for the last month or two. It also
previously included a fair amount of legacy weight that was no longer
relevant.
* Add missing secret insertions
* Improve logic for deployments
This changes the logic for the deployments so that pushes to 'stable/*'
no longer trigger any deployment to qiskit.org. Instead, tag events
trigger a deployment to the relevant stable branch, and a tag event of
the _latest_ tag triggers a deployment to the documentation root.
The translatables logic is modified to push only the latest full-release
tag.
* Add no-optionals and full-optionals test runs
This restructures the CI slightly to perform a complete "no-optionals"
run, and a complete "all optionals" run (for the optionals that are
readily accessible as Python packages, without complex additional
setup). Previously, we only did a partial test with some of the oldest
optional components, which could have allowed for behaviour to
accidentally require optional components without us noticing.
This splits the `requirements-dev.txt` file into two; lines that remain
in the file are what is actually _required_ to run the test suite,
run the style checks, and do the documentation build. The rest (and
everything that was missing) is added to a new
`requirements-optional.txt` file, which can be used to pull in (almost)
all of the packages that Terra can use to provide additional /
accelerated functionality.
Several tests needed to gain additional skips to account for this
change. There is a good chance that some tests are missing skips for
some libraries that are not the first point of failure, but it's hard to
test explicitly for these in one go.
* Fix typo in coverage workflow
* Try relaxing ipython constraints
* Squash newly exposed lint failures
* Fix typo in tutorials pipeline
* Update the 'also update' comments
* Remove unneeded qiskit-toqm dependency
* Section requirements-optional.txt
* Test all optionals on min not max Python version
Optionals are generally more likely to have been made available on the
older Pythons, and some may take excessively long to provide wheels for
the latest versions of Python.
* Add missing test skip
* Fix optional call
* Use correct boolean syntax
* Fix tests relying on Jupyter
* Install ipykernel in tutorials
* Remove HAS_PDFLATEX skip from quantum_info tests
For simple LaTeX tests, IPython/Jupyter can handle the compilation
internally using MathJax, and doesn't actually need a `pdflatex`
installation. We only need that when we're depending on LaTeX libraries
that are beyond what MathJax can handle natively.
* Include additional tutorials dependencies
* Install all of Terra's optionals in tutorial run
* Do not install all optionals in docs build
* Use class-level skips where appropriate
* Do not install ibmq-provider in tutorials run
* Include matplotlib in docs requirements
* Remove unnecessary whitespace
* Split long pip line
* Only install graphviz when installOptionals
Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>
* Install visualization extras in docs build
* Don't `--upgrade` when installing optionals
This is to prevent any optionals that have a dependency on Terra from
potentially upgrading it to some PyPI version during their installation.
This shouldn't happen with the current development model of having only
one supported stable branch and the main branch, but the `-U` is
unnecessary, and not having it is safer for the future.
* Update secondary installation of Terra in docs build
* Install all optionals during the docs build
I yoyo-ed on this, not wanting it to be too onerous to build the
documentation, but in the end, we need to have the optional features
installed in order to build most of the documentation of those features,
and some unrelated parts of the documentation may use these features to
enhance their own output.
* Fix test setup job
* Remove duplication of matplotlib in requirements files
* Update image test installation command
* Restore editable build
* Move `pip check` to `pip` section
* Remove redundant "post-install" description
* Expand comment on first-stage choices
* pytohn lol
---------
Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>
Ideally we'd be running the Neko tests against the requirements that
will actually form the complete package. In practice, however, there
are persistent failures from Neko stemming from PySCF's use of SciPy,
which we get via Nature, which is no longer under Qiskit control.
While we work out what the future of the Neko test suite is, with the
application modules now no longer being IBM-managed Qiskit packages, we
can apply the CI constraints so we can return to getting reports from
the parts of the integration tests that are actually within our control.
* Pin Aer to known good version
The release of Aer 0.12.1 included a bug in the results output that made
the general `Result` object unpickleable. Several places in our tests
assume that these objects should be able to be pickled to be sent across
a process boundary, or such like.
* Make GitHub Actions respect constraints
* Use stable Python C API for building Rust extension
This commit tweaks the rust extension code to start using the PyO3 abi3
flag to build binaries that are compatible with all python versions, not
just a single release. Previously, we were building against the version
specific C API and that resulted in needing abinary file for each
supported python version on each supported platform/architecture. By
using the abi3 feature flag and marking the wheels as being built with
the limited api we can reduce our packaging overhead to just having one
wheel file per supported platform/architecture.
The only real code change needed here was to update the memory
marginalization function. PyO3's abi3 feature is incompatible with
returning a big int object from rust (the C API they use for that
conversion isn't part of the stable C API). So this commit updates the
function to convert to create a python int manually using the PyO3 api
where that was being done before.
Co-authored-by: Jake Lishman <jake.lishman@ibm.com>
* Set minimum version on abi3 flag to Python 3.8
* Fix lint
* Use py_limited_api="auto" on RustExtension
According to the docs for the setuptools-rust RustExtension class:
https://setuptools-rust.readthedocs.io/en/latest/reference.html#setuptools_rust.RustExtension
The best setting to use for the py_limited_api argument is `"auto"` as
this will use the setting in the PyO3 module to determine the correct
value to set. This commit updates the setup.py to follow the
recommendation in the docs.
* Update handling of phase input to expval rust calls
The pauli_expval module in Rust that Statevector and DensityMatrix
leverage when computing defines the input type of the phase argument as
Complex64. Previously, the quantum info code in the Statevector and
DensityMatrix classes were passing in a 1 element ndarray for this
parameter. When using the the version specific Python C API in pyo3 it
would convert the single element array to a scalar value. However when
using abi3 this handling was not done (or was not done correctly) and
this caused the tests to fail. This commit updates the quantum info
module to pass the phase as a complex value instead of a 1 element numpy
array to bypass this behavior change in PyO3 when using abi3.
Co-authored-by: Jake Lishman <jake.lishman@ibm.com>
* Set py_limited_api explicitly to True
* DNM: Test cibuildwheel works with abi3
* Add abi3audit to cibuildwheel repair step
* Force setuptools to use abi3 tag
* Add wheel to sdist build
* Workaround abiaudit3 not moving wheels and windows not having a default repair command
* Add source of setup.py hack
* Add comment about pending pyo3 abi3 bigint support
* Revert "DNM: Test cibuildwheel works with abi3"
This reverts commit 8ca24cf1e4.
* Add release note
* Simplify setting abi3 tag in built wheels
* Update releasenotes/notes/use-abi3-4a935e0557d3833b.yaml
Co-authored-by: Jake Lishman <jake@binhbar.com>
* Update release note
* Update releasenotes/notes/use-abi3-4a935e0557d3833b.yaml
---------
Co-authored-by: Jake Lishman <jake.lishman@ibm.com>
Co-authored-by: Jake Lishman <jake@binhbar.com>
* Drop support for Python 3.7
This updates our documentation and build processes to set Python 3.8 as
the oldest support version of Python, consistent with our policy on
Python support.
This commit also removes remaining Python-version gating, since we were
principally using this to account for differences between 3.7 and 3.8.
Several backport packages are no longer required for any suppported
Python version, because of this change.
* Remove test for multiprocessing on Mac
With Python 3.7 support now dropped, there are no versions of Python on
macOS that we expect to support with multiprocessing again. There's no
guarantee that the hypothetical Python 10.11 (???) that this test was
previously checking would be any more valid than existing versions.
---------
Co-authored-by: Matthew Treinish <mtreinish@kortar.org>
* Upgrade straggler GitHub Actions to Node 16
The Node 12 runners are due to be removed from GitHub Actions this
summer. We updated most of them in 5d977fbf (gh-8856), but a couple
remained un-updated.
The `actions-rs/toolchain` action is unmaintained. The replacement used
in this commit is actively updated, and has mostly the same interface,
although the Rust version is typically specified using the action
version.
* Reinstate Rust toolchain override
On some OSes and configurations (usually Windows), there can be problems
when a binary attempts to one that is in use, especially itself. For
this reason, it is more reliable to use `python -m pip install ...` than
`pip install`. We have seen some CI failures on Windows due to `pip`
failing to update itself because of the direct-executable `pip install`
form.
With our switch to using the GitHub merge queue with speculative
merging, it's now possible for us to have two push events to a protected
branch (like `main`) within a couple of minutes. If this happens, the
eager cancellation rules we have set for the coverage and Neko runs in
GHA trigger, causing a spurious build failure to appear in the commit
logs, and for the coverage history to be lost for one commit.
Instead, we can set the cancel-on-update behaviour to only trigger on PR
sync events, not branch-push events.
Up-to-date Python packages should not require this step, however there
are several packages, especially those that are optional for Terra
functionality, that do not yet contain `pyproject.toml` files when
building them from source. In these cases, `pip` will begin erroring
out from version 23.1 if `wheel` is not installed.
This commit proactively ensures that the minimum build dependencies for
legacy Python packages is prepared and up-to-date before attempting
installations; this includes ensuring that these are updated _inside_
any created venvs as well as outside them.
dependabot[bot]