Autoformat TypeScript with Prettier (#76)

Relates to https://github.com/Qiskit/documentation/issues/6, which is
about docs. This PR only adds Prettier to our support code. We might
want to extend Prettier to docs in a follow up.

Having a consistent format is convenient and saves us time so that
humans don't have to manually move things around.

We have a non-trivial amount of TypeScript tooling, so this is worth
adding imo.
This commit is contained in:
Eric Arellano 2023-10-20 11:35:35 -04:00 committed by GitHub
parent 5e70749d29
commit fdfc2e5e2e
35 changed files with 814 additions and 629 deletions

View File

@ -41,4 +41,4 @@ body:
- type: markdown
attributes:
value: Thank you for your feedback! If you are interested in joining the IBM Quantum Feedback Program, <a href="https://www.ibm.com/quantum/feedback-program">sign up here</a>.
value: Thank you for your feedback! If you are interested in joining the IBM Quantum Feedback Program, <a href="https://www.ibm.com/quantum/feedback-program">sign up here</a>.

View File

@ -5,4 +5,4 @@ contact_links:
about: Open an issue in the Qiskit repository for the docs found at https://docs.quantum-computing.ibm.com/api/qiskit.
- name: Qiskit Runtime Client API feedback
url: https://github.com/Qiskit/qiskit-ibm-runtime/issues/new/choose
about: Open an issue in the qiskit-ibm-runtime repository for the docs found at https://docs.quantum-computing.ibm.com/api/qiskit-ibm-runtime/runtime_service.
about: Open an issue in the qiskit-ibm-runtime repository for the docs found at https://docs.quantum-computing.ibm.com/api/qiskit-ibm-runtime/runtime_service.

View File

@ -42,4 +42,4 @@ body:
- type: markdown
attributes:
value: Thank you for your feedback! If you are interested in joining the IBM Quantum Feedback Program, <a href="https://www.ibm.com/quantum/feedback-program">sign up here</a>.
value: Thank you for your feedback! If you are interested in joining the IBM Quantum Feedback Program, <a href="https://www.ibm.com/quantum/feedback-program">sign up here</a>.

View File

@ -23,10 +23,10 @@ body:
attributes:
label: Please describe the UI problem.
description: >
Please be as specific as possible. Include steps to reproduce, the operating system and browser you're using, a screenshot if possible, etc.
Please be as specific as possible. Include steps to reproduce, the operating system and browser you're using, a screenshot if possible, etc.
validations:
required: true
- type: markdown
attributes:
value: Thank you for your feedback! If you are interested in joining the IBM Quantum Feedback Program, <a href="https://www.ibm.com/quantum/feedback-program">sign up here</a>.
value: Thank you for your feedback! If you are interested in joining the IBM Quantum Feedback Program, <a href="https://www.ibm.com/quantum/feedback-program">sign up here</a>.

View File

@ -31,4 +31,3 @@ body:
- type: markdown
attributes:
value: Thank you for your feedback! If you are interested in joining the IBM Quantum Feedback Program, <a href="https://www.ibm.com/quantum/feedback-program">sign up here</a>.

View File

@ -32,6 +32,8 @@ jobs:
run: npm run check:spelling
- name: Link checker
run: npm run check:links
- name: Formatting
run: npm run check:fmt
- name: Typecheck
run: npm run typecheck
- name: Run infrastructure tests

1
.prettierignore Normal file
View File

@ -0,0 +1 @@
docs

View File

@ -1,9 +1,10 @@
<!-- Copyright Contributors to the Qiskit project. -->
# Code of Conduct
All members of this project agree to adhere to the Qiskit Code of Conduct listed at [https://github.com/Qiskit/qiskit/blob/master/CODE_OF_CONDUCT.md](https://github.com/Qiskit/qiskit/blob/master/CODE_OF_CONDUCT.md)
----
---
License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/),
Copyright Contributors to Qiskit.
Copyright Contributors to Qiskit.

View File

@ -3,11 +3,13 @@
The documentation content home for https://docs.quantum-computing.ibm.com. Note this repo does not contain content for https://learning.quantum-computing.ibm.com/ or https://qiskit.org. Refer to https://github.com/Qiskit/qiskit to make changes to the docs at https://qiskit.org/documentation.
# Improving IBM Quantum & Qiskit Documentation
Maintaining up-to-date documentation is a huge challenge for any software project, especially in a field like quantum computing where the pace at which advances in new research and technological capabilities happen incredibly fast. As a result, we greatly appreciate any who take the time to support us in keeping this content accurate and up to the highest quality standard possible to benefit the broadest range of users.
Read on for more information about how to support this project:
## Best ways to contribute to Documentation
### 1. Report bugs, inaccuracies or general content issues
This is the quickest, easiest, and most helpful way to contribute to this project and improve the quality of Qiskit and IBM Quantum documentation. There are a few different ways to report issues, depending on where it was found:
@ -18,7 +20,7 @@ This is the quickest, easiest, and most helpful way to contribute to this projec
### 2. Suggest new content
If you think there are gaps in our documentation, or sections that could be expanded upon, we invite you to open a new content request issue [here](https://github.com/Qiskit/documentation/issues/new/choose).
If you think there are gaps in our documentation, or sections that could be expanded upon, we invite you to open a new content request issue [here](https://github.com/Qiskit/documentation/issues/new/choose).
Not every new content suggestion is a good fit for docs, nor are we able to prioritize every request immediately. However, we will do our best to respond to content requests in a timely manner, and we greatly appreciate our community's efforts in generating new ideas.
@ -30,27 +32,27 @@ Please note: we DO NOT accept unsolicited PRs for new pages or large updates to
You can help the team prioritize already-open issues by doing the following:
- For bug reports, leave a comment in the issue if you have also been experiencing the same problem and can reproduce it (include as much information as you can, e.g., browser type, Qiskit version, etc.).
- For bug reports, leave a comment in the issue if you have also been experiencing the same problem and can reproduce it (include as much information as you can, e.g., browser type, Qiskit version, etc.).
- For new content requests, leave a comment or upvote (👍) in the issue if you also would like to see that new content added.
### 4. Fix an open issue
You can look through the open issues we have in this repo and address them with a PR. We recommend focusing on issues with the "good first issue" label.
You can look through the open issues we have in this repo and address them with a PR. We recommend focusing on issues with the "good first issue" label.
Before getting started on an issue, remember to do the following:
1. Read the [Code of Conduct](https://github.com/Qiskit/documentation/blob/main/CODE_OF_CONDUCT.md)
1. Read the [Code of Conduct](https://github.com/Qiskit/documentation/blob/main/CODE_OF_CONDUCT.md)
2. Check for open, unassigned issues with the "good first issue" label
3. Select an issue that is not already assigned to someone and leave a comment to request to be assigned
Once you have an issue to work on, see the "How to work with this repo" section below to get going, then open a PR.
Before opening a PR, remember to do the following:
1. Check that you have addressed all the requirements from the original issue
2. Run the spell checker
3. Use the GitHub "fixes" notation to [link your PR to the issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue) you are addressing
# How to work with this repo
## Prerequisites to building the docs locally
@ -61,7 +63,7 @@ First, install the below software:
1. [Node.js](https://nodejs.org/en). If you expect to use JavaScript in other projects, consider using [NVM](https://github.com/nvm-sh/nvm). Otherwise, consider using [Homebrew](https://formulae.brew.sh/formula/node) or installing [Node.js directly](https://nodejs.org/en).
2. [Docker](https://www.docker.com). You must also ensure that it is running.
* You can use [Colima](https://github.com/abiosoft/colima) or [Rancher Desktop](https://rancherdesktop.io). When installing Rancher Desktop, choose Moby/Dockerd as the engine, rather than nerdctl. To ensure it's running, open up the app "Rancher Desktop".
- You can use [Colima](https://github.com/abiosoft/colima) or [Rancher Desktop](https://rancherdesktop.io). When installing Rancher Desktop, choose Moby/Dockerd as the engine, rather than nerdctl. To ensure it's running, open up the app "Rancher Desktop".
Then, install the dependencies with:
@ -75,11 +77,11 @@ Run `./start` in your terminal, then open http://localhost:3000/start in your br
The local preview does not include the initial index page and the top nav bar from docs.quantum-computing.ibm.com. Therefore, you must directly navigate in the URL to the folder that you want:
* http://localhost:3000/build
* http://localhost:3000/start
* http://localhost:3000/run
* http://localhost:3000/transpile
* http://localhost:3000/verify
- http://localhost:3000/build
- http://localhost:3000/start
- http://localhost:3000/run
- http://localhost:3000/transpile
- http://localhost:3000/verify
## Preview the docs in PRs
@ -125,7 +127,7 @@ We use [cSpell](https://cspell.org) to check for spelling. The `lint` job in CI
There are two ways to check spelling locally, rather than needing CI.
1.
1.
```bash
# Only check spelling
@ -154,3 +156,9 @@ Ayyyyy, this is a fake description.
2. Add the word to the file `cSpell.json` in the `words` section. The word is not case-sensitive.
If the word appears in multiple files, prefer the second approach to add it to `cSpell.json`.
## Format files
Run `npm run fmt` to automatically format MDX files.
To check that formatting is valid without actually making changes, run `npm run check:fmt` or `npm run check`.

16
package-lock.json generated
View File

@ -33,6 +33,7 @@
"mkdirp": "^3.0.1",
"p-map": "^6.0.0",
"p-queue": "^7.4.1",
"prettier": "^3.0.3",
"rehype-parse": "^8.0.0",
"rehype-remark": "^9.1.2",
"remark-gfm": "^3.0.1",
@ -7713,6 +7714,21 @@
"node": ">=8"
}
},
"node_modules/prettier": {
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/prettier/-/prettier-3.0.3.tgz",
"integrity": "sha512-L/4pUDMxcNa8R/EthV08Zt42WBO4h1rarVtK0K+QJG0X187OLo7l699jWw0GKuwzkPQ//jMFA/8Xm6Fh3J/DAg==",
"dev": true,
"bin": {
"prettier": "bin/prettier.cjs"
},
"engines": {
"node": ">=14"
},
"funding": {
"url": "https://github.com/prettier/prettier?sponsor=1"
}
},
"node_modules/pretty-format": {
"version": "29.7.0",
"resolved": "https://registry.npmjs.org/pretty-format/-/pretty-format-29.7.0.tgz",

View File

@ -5,10 +5,12 @@
"author": "Qiskit Development Team",
"license": "Apache-2.0",
"scripts": {
"check": "npm run check:metadata && npm run check:spelling && npm run check:links",
"check": "npm run check:metadata && npm run check:spelling && npm run check:links && npm run check:fmt",
"check:metadata": "node -r esbuild-register scripts/commands/checkMetadata.ts",
"check:links": "node -r esbuild-register scripts/commands/checkLinks.ts",
"check:spelling": "cspell --relative --no-progress docs/**/*.md*",
"check:fmt": "prettier --check .",
"fmt": "prettier --write .",
"test": "jest",
"typecheck": "tsc"
},
@ -33,6 +35,7 @@
"mkdirp": "^3.0.1",
"p-map": "^6.0.0",
"p-queue": "^7.4.1",
"prettier": "^3.0.3",
"rehype-parse": "^8.0.0",
"rehype-remark": "^9.1.2",
"remark-gfm": "^3.0.1",

View File

@ -11,73 +11,69 @@
// that they have been altered from the originals.
import { globby } from "globby";
import { existsSync, readFileSync } from 'fs';
import path from 'node:path';
import markdownLinkExtractor from 'markdown-link-extractor';
import { existsSync, readFileSync } from "fs";
import path from "node:path";
import markdownLinkExtractor from "markdown-link-extractor";
const DOCS_ROOT = "./docs"
const CONTENT_FILE_EXTENSIONS = [".md", ".mdx", ".ipynb"]
const DOCS_ROOT = "./docs";
const CONTENT_FILE_EXTENSIONS = [".md", ".mdx", ".ipynb"];
const IGNORE_LIST = [
'docs/run/instances.mdx',
'docs/start/index.mdx',
]
const IGNORE_LIST = ["docs/run/instances.mdx", "docs/start/index.mdx"];
class Link {
readonly value: string
readonly anchor: string
readonly origin: string
readonly isExternal: boolean
readonly value: string;
readonly anchor: string;
readonly origin: string;
readonly isExternal: boolean;
constructor(linkString: string, origin: string) {
/*
/*
* linkString: Link as it appears in source file
* origin: Path to source file containing link
*/
const splitLink = linkString.split('#', 1)
this.value = splitLink[0]
this.anchor = (splitLink.length > 1) ? `#${splitLink[1]}` : ''
this.origin = origin
this.isExternal = linkString.startsWith("http")
const splitLink = linkString.split("#", 1);
this.value = splitLink[0];
this.anchor = splitLink.length > 1 ? `#${splitLink[1]}` : "";
this.origin = origin;
this.isExternal = linkString.startsWith("http");
}
resolve(): string[] {
/*
* Return list of possible paths link could resolve to
*/
if ( this.isExternal ) { return [ this.value ] }
if ( this.value === '' ) { return [ this.origin ] } // link is just anchor
if ( this.value.startsWith("/images") ) {
return [ path.join("public/", this.value) ]
if (this.isExternal) {
return [this.value];
}
if (this.value === "") {
return [this.origin];
} // link is just anchor
if (this.value.startsWith("/images")) {
return [path.join("public/", this.value)];
}
let baseFilePath
if (this.value.startsWith('/')) {
let baseFilePath;
if (this.value.startsWith("/")) {
// Path is relative to DOCS_ROOT
baseFilePath = path.join(DOCS_ROOT, this.value)
baseFilePath = path.join(DOCS_ROOT, this.value);
} else {
// Path is relative to origin file
baseFilePath = path.join(
path.dirname(this.origin),
this.value
)
baseFilePath = path.join(path.dirname(this.origin), this.value);
}
// Remove trailing '/' from path.join
baseFilePath = baseFilePath.replace(/\/$/gm, '')
baseFilePath = baseFilePath.replace(/\/$/gm, "");
// File may have one of many extensions (.md, .ipynb etc.), and/or be
// directory with an index file (e.g. `docs/build` should resolve to
// `docs/build/index.mdx`). We return a list of possible filenames.
let possibleFilePaths = []
for (let index of ['', '/index']) {
let possibleFilePaths = [];
for (let index of ["", "/index"]) {
for (let extension of CONTENT_FILE_EXTENSIONS) {
possibleFilePaths.push(
baseFilePath + index + extension
)
possibleFilePaths.push(baseFilePath + index + extension);
}
}
return possibleFilePaths
return possibleFilePaths;
}
check(filePathCache: string[] = []): boolean {
@ -87,61 +83,68 @@ class Link {
*/
if (this.isExternal) {
// External link checking not supported yet
return true
return true;
}
const possiblePaths = this.resolve()
const possiblePaths = this.resolve();
for (let filePath of possiblePaths) {
if (filePathCache.includes(filePath)) {
return true
return true;
}
}
// Check disk for files not in cache (images etc.)
for (let filePath of possiblePaths) {
if (existsSync(filePath)) {
return true
return true;
}
}
console.log(`${this.origin}: Could not find link '${this.value}'`)
return false
console.log(`${this.origin}: Could not find link '${this.value}'`);
return false;
}
}
function markdownFromNotebook(source: string): string {
let markdown = ''
let markdown = "";
for (let cell of JSON.parse(source).cells) {
if (cell.source === 'markdown') {
markdown += cell.source
if (cell.source === "markdown") {
markdown += cell.source;
}
}
return markdown
return markdown;
}
function checkLinksInFile(filePath: string, filePaths: string[]): boolean {
if (filePath.startsWith("docs/api")) { return true }
if (IGNORE_LIST.includes(filePath)) { return true }
const source = readFileSync(filePath, {encoding: 'utf8'})
const markdown = (path.extname(filePath) === '.ipynb') ? markdownFromNotebook(source) : source
const links = markdownLinkExtractor(source).links.map((x: string) => new Link(x, filePath))
let allGood = true
for (let link of links) {
allGood = link.check(filePaths) && allGood
if (filePath.startsWith("docs/api")) {
return true;
}
return allGood
if (IGNORE_LIST.includes(filePath)) {
return true;
}
const source = readFileSync(filePath, { encoding: "utf8" });
const markdown =
path.extname(filePath) === ".ipynb" ? markdownFromNotebook(source) : source;
const links = markdownLinkExtractor(source).links.map(
(x: string) => new Link(x, filePath),
);
let allGood = true;
for (let link of links) {
allGood = link.check(filePaths) && allGood;
}
return allGood;
}
async function main() {
const filePaths = await globby('docs/**/*.{ipynb,md,mdx}')
let allGood = true
const filePaths = await globby("docs/**/*.{ipynb,md,mdx}");
let allGood = true;
for (let sourceFile of filePaths) {
allGood = checkLinksInFile(sourceFile, filePaths) && allGood
allGood = checkLinksInFile(sourceFile, filePaths) && allGood;
}
if (!allGood) {
console.log("\nSome links appear broken 💔\n")
process.exit(1)
console.log("\nSome links appear broken 💔\n");
process.exit(1);
}
}
main()
main();

View File

@ -1,9 +1,9 @@
type LinkExtractionResult = {
links: string[];
anchors: string[];
}
};
declare module 'markdown-link-extractor' {
declare module "markdown-link-extractor" {
function markdownLinkExtractor(string): LinkExtractionResult;
export = markdownLinkExtractor;
}

View File

@ -21,27 +21,27 @@
//
// Pass `--packages {qiskit,qiskit-ibm-provider,qiskit-ibm-runtime} to only generate for certain projects.
import { $ } from 'zx';
import { zxMain } from '../lib/zx';
import { getRequiredEnv } from '../lib/env';
import { GithubApiClient } from '../lib/GithubApiClient';
import { pathExists, getRoot } from '../lib/fs';
import { readFile, writeFile } from 'fs/promises';
import { globby } from 'globby';
import { join, parse, relative } from 'path';
import { sphinxHtmlToMarkdown } from '../lib/sphinx/sphinxHtmlToMarkdown';
import { first, last, uniq, uniqBy } from 'lodash';
import { mkdirp } from 'mkdirp';
import { WebCrawler } from '../lib/WebCrawler';
import { downloadImages } from '../lib/downloadImages';
import { generateToc } from '../lib/sphinx/generateToc';
import { SphinxToMdResult } from '../lib/sphinx/SphinxToMdResult';
import { mergeClassMembers } from '../lib/sphinx/mergeClassMembers';
import { flatFolders } from '../lib/sphinx/flatFolders';
import { updateLinks } from '../lib/sphinx/updateLinks';
import { addFrontMatter } from '../lib/sphinx/addFrontMatter';
import { dedupeResultIds } from '../lib/sphinx/dedupeIds';
import { removePrefix, removeSuffix } from '../lib/stringUtils';
import { $ } from "zx";
import { zxMain } from "../lib/zx";
import { getRequiredEnv } from "../lib/env";
import { GithubApiClient } from "../lib/GithubApiClient";
import { pathExists, getRoot } from "../lib/fs";
import { readFile, writeFile } from "fs/promises";
import { globby } from "globby";
import { join, parse, relative } from "path";
import { sphinxHtmlToMarkdown } from "../lib/sphinx/sphinxHtmlToMarkdown";
import { first, last, uniq, uniqBy } from "lodash";
import { mkdirp } from "mkdirp";
import { WebCrawler } from "../lib/WebCrawler";
import { downloadImages } from "../lib/downloadImages";
import { generateToc } from "../lib/sphinx/generateToc";
import { SphinxToMdResult } from "../lib/sphinx/SphinxToMdResult";
import { mergeClassMembers } from "../lib/sphinx/mergeClassMembers";
import { flatFolders } from "../lib/sphinx/flatFolders";
import { updateLinks } from "../lib/sphinx/updateLinks";
import { addFrontMatter } from "../lib/sphinx/addFrontMatter";
import { dedupeResultIds } from "../lib/sphinx/dedupeIds";
import { removePrefix, removeSuffix } from "../lib/stringUtils";
import yargs from "yargs/yargs";
import { hideBin } from "yargs/helpers";
@ -61,84 +61,94 @@ type Pkg = {
collapsed?: boolean;
nestModule?(id: string): boolean;
};
transformLink?: (url: string, text?: string) => { url: string; text?: string } | undefined;
transformLink?: (
url: string,
text?: string,
) => { url: string; text?: string } | undefined;
};
type PkgHtml = { pkg: Pkg; version: string; path: string };
const PACKAGES: Pkg[] = [
{
title: 'Qiskit Runtime IBM Client',
name: 'qiskit-ibm-runtime',
githubSlug: 'qiskit/qiskit-ibm-runtime',
title: "Qiskit Runtime IBM Client",
name: "qiskit-ibm-runtime",
githubSlug: "qiskit/qiskit-ibm-runtime",
baseUrl: `https://qiskit.org/ecosystem/ibm-runtime`,
initialUrls: [
`https://qiskit.org/ecosystem/ibm-runtime/apidocs/ibm-runtime.html`,
],
transformLink(url, text) {
const prefixes = [
'https://qiskit.org/documentation/apidoc/',
'https://qiskit.org/documentation/stubs/',
"https://qiskit.org/documentation/apidoc/",
"https://qiskit.org/documentation/stubs/",
];
let updateText = url === text;
const prefix = prefixes.find((prefix) => url.startsWith(prefix));
if (prefix) {
url = removePrefix(url, prefix);
url = removeSuffix(url, '.html');
url = removeSuffix(url, ".html");
const newText = updateText ? url : undefined;
return { url: `/api/qiskit/${url}`, text: newText };
}
},
},
{
title: 'Qiskit IBM Provider',
name: 'qiskit-ibm-provider',
githubSlug: 'qiskit/qiskit-ibm-provider',
title: "Qiskit IBM Provider",
name: "qiskit-ibm-provider",
githubSlug: "qiskit/qiskit-ibm-provider",
baseUrl: `https://qiskit.org/ecosystem/ibm-provider`,
initialUrls: [`https://qiskit.org/ecosystem/ibm-provider/apidocs/ibm-provider.html`],
initialUrls: [
`https://qiskit.org/ecosystem/ibm-provider/apidocs/ibm-provider.html`,
],
transformLink(url, text) {
const prefixes = [
'https://qiskit.org/documentation/apidoc/',
'https://qiskit.org/documentation/stubs/',
"https://qiskit.org/documentation/apidoc/",
"https://qiskit.org/documentation/stubs/",
];
let updateText = url === text;
const prefix = prefixes.find((prefix) => url.startsWith(prefix));
if (prefix) {
url = removePrefix(url, prefix);
url = removeSuffix(url, '.html');
url = removeSuffix(url, ".html");
const newText = updateText ? url : undefined;
return { url: `/api/qiskit/${url}`, text: newText };
}
},
},
{
title: 'Qiskit',
name: 'qiskit',
githubSlug: 'qiskit/qiskit',
title: "Qiskit",
name: "qiskit",
githubSlug: "qiskit/qiskit",
baseUrl: `https://qiskit.org/documentation`,
initialUrls: [`https://qiskit.org/documentation/apidoc/index.html`],
ignore(id: string): boolean {
return id.startsWith('qiskit.opflow') || id.startsWith('qiskit.algorithms');
return (
id.startsWith("qiskit.opflow") || id.startsWith("qiskit.algorithms")
);
},
tocOptions: {
collapsed: true,
nestModule(id: string) {
return id.split('.').length > 2;
return id.split(".").length > 2;
},
},
transformLink(url) {
// Transform links from ignored modules
let path = last(url.split('/'))!;
if (path.includes('#')) {
path = path.split('#').join('.html#');
let path = last(url.split("/"))!;
if (path.includes("#")) {
path = path.split("#").join(".html#");
} else {
path += '.html';
path += ".html";
}
if (path?.startsWith('algorithms') || path?.startsWith('opflow')) {
if (path?.startsWith("algorithms") || path?.startsWith("opflow")) {
return { url: `http://qiskit.org/documentation/apidoc/${path}` };
}
if (path?.startsWith('qiskit.algorithms.') || path?.startsWith('qiskit.opflow.')) {
if (
path?.startsWith("qiskit.algorithms.") ||
path?.startsWith("qiskit.opflow.")
) {
return { url: `http://qiskit.org/documentation/stubs/${path}` };
}
},
@ -155,8 +165,8 @@ const readArgs = (): Arguments => {
choices: pkgs,
description: "What packages to update",
})
.parseSync()
};
.parseSync();
};
zxMain(async () => {
const args = readArgs();
@ -177,7 +187,11 @@ zxMain(async () => {
continue;
}
await downloadHtml({ baseUrl: pkg.baseUrl, initialUrls: pkg.initialUrls, destination });
await downloadHtml({
baseUrl: pkg.baseUrl,
initialUrls: pkg.initialUrls,
destination,
});
}
for (const src of pkgHtmls) {
@ -203,7 +217,7 @@ async function getLatestVersion(githubSlug: string): Promise<string> {
const releases = await github.getReleases({ slug: githubSlug });
const latestVersion = first(releases)?.tag_name;
if (!latestVersion) throw new Error('Cannot fetch latest version');
if (!latestVersion) throw new Error("Cannot fetch latest version");
return latestVersion;
}
@ -239,24 +253,30 @@ async function downloadHtml(options: {
},
});
await crawler.run();
console.log(`Download summary from ${baseUrl}`, { success: successCount, error: errorCount });
console.log(`Download summary from ${baseUrl}`, {
success: successCount,
error: errorCount,
});
}
async function convertHtmlToMarkdown(
htmlPath: string,
markdownPath: string,
baseSourceUrl: string,
pkg: PkgHtml
pkg: PkgHtml,
) {
const files = await globby(['apidocs/**.html', 'apidoc/**.html', 'stubs/**.html'], {
cwd: htmlPath,
});
const files = await globby(
["apidocs/**.html", "apidoc/**.html", "stubs/**.html"],
{
cwd: htmlPath,
},
);
const ignore = pkg.pkg.ignore ?? (() => false);
let results: Array<SphinxToMdResult & { url: string }> = [];
for (const file of files) {
const html = await readFile(join(htmlPath, file), 'utf-8');
const html = await readFile(join(htmlPath, file), "utf-8");
const result = await sphinxHtmlToMarkdown({
html,
url: `${pkg.pkg.baseUrl}/${file}`,
@ -273,10 +293,12 @@ async function convertHtmlToMarkdown(
const allImages = uniqBy(
results.flatMap((result) => result.images),
(image) => image.src
(image) => image.src,
);
const dirsNeeded = uniq(results.map((result) => parse(urlToPath(result.url)).dir));
const dirsNeeded = uniq(
results.map((result) => parse(urlToPath(result.url)).dir),
);
for (const dir of dirsNeeded) {
await mkdirp(dir);
}
@ -291,7 +313,7 @@ async function convertHtmlToMarkdown(
await writeFile(urlToPath(result.url), result.markdown);
}
console.log('Generating toc');
console.log("Generating toc");
const toc = generateToc({
pkg: {
title: pkg.pkg.title,
@ -302,11 +324,17 @@ async function convertHtmlToMarkdown(
},
results,
});
await writeFile(`${markdownPath}/_toc.json`, JSON.stringify(toc, null, 2) + '\n');
await writeFile(
`${markdownPath}/_toc.json`,
JSON.stringify(toc, null, 2) + "\n",
);
console.log('Downloading images');
console.log("Downloading images");
await downloadImages(
allImages.map((img) => ({ src: img.src, dest: `${getRoot()}/public${img.dest}` }))
allImages.map((img) => ({
src: img.src,
dest: `${getRoot()}/public${img.dest}`,
})),
);
}

View File

@ -10,12 +10,12 @@
// copyright notice, and modified files need to carry a notice indicating
// that they have been altered from the originals.
import { merge } from 'lodash';
import { merge } from "lodash";
export class GithubApiClient {
token: string;
constructor(options: { token: string}) {
constructor(options: { token: string }) {
this.token = options.token;
}
@ -24,9 +24,8 @@ export class GithubApiClient {
return this.fetch<GithubRelease[]>(`repos/${slug}/releases`);
}
private getUrl(url: string) {
if (url.startsWith('https:')) return url;
if (url.startsWith("https:")) return url;
return `https://api.github.com/${url}`;
}
@ -37,11 +36,11 @@ export class GithubApiClient {
{
headers: {
Authorization: `token ${this.token}`,
Accept: 'application/vnd.github.v3+json',
Accept: "application/vnd.github.v3+json",
},
},
options
)
options,
),
);
if (!response.ok) {
console.error(response);

View File

@ -10,8 +10,8 @@
// copyright notice, and modified files need to carry a notice indicating
// that they have been altered from the originals.
import { load } from 'cheerio';
import PQueue from 'p-queue';
import { load } from "cheerio";
import PQueue from "p-queue";
export class WebCrawler {
private initialUrls: string[];
@ -43,12 +43,12 @@ export class WebCrawler {
this.onError = options.onError;
this.onSkip = options.onSkip;
this.queue = new PQueue({ concurrency: 4 });
this.linkSelector = options.linkSelector ?? 'a';
this.linkSelector = options.linkSelector ?? "a";
}
async run() {
return new Promise((resolve) => {
this.queue.once('idle', resolve);
this.queue.once("idle", resolve);
this.queueUrls(this.initialUrls);
});
}
@ -78,8 +78,8 @@ export class WebCrawler {
}
if (response.ok) {
if (!response.headers.get('content-type')?.includes('text/html')) {
this.onSkip?.(url, 'Not html');
if (!response.headers.get("content-type")?.includes("text/html")) {
this.onSkip?.(url, "Not html");
return;
}
@ -105,11 +105,11 @@ export class WebCrawler {
.toArray()
.flatMap((el) => {
const $el = $(el);
const href = $el.attr('href');
const href = $el.attr("href");
if (!href) return [];
const parsedUrl = new URL(href, url);
parsedUrl.hash = '';
parsedUrl.hash = "";
return [parsedUrl.toString()];
});
return links;

View File

@ -10,15 +10,17 @@
// copyright notice, and modified files need to carry a notice indicating
// that they have been altered from the originals.
import { pathExists } from './fs';
import { mkdirp } from 'mkdirp';
import { dirname } from 'path';
import { createWriteStream } from 'node:fs';
import { finished } from 'stream/promises';
import { Readable } from 'stream';
import pMap from 'p-map';
import { pathExists } from "./fs";
import { mkdirp } from "mkdirp";
import { dirname } from "path";
import { createWriteStream } from "node:fs";
import { finished } from "stream/promises";
import { Readable } from "stream";
import pMap from "p-map";
export async function downloadImages(images: Array<{ src: string; dest: string }>) {
export async function downloadImages(
images: Array<{ src: string; dest: string }>,
) {
await pMap(
images,
async (img) => {
@ -32,6 +34,6 @@ export async function downloadImages(images: Array<{ src: string; dest: string }
console.log(`Error downloading ${img.src} to ${img.dest}`);
}
},
{ concurrency: 4 }
{ concurrency: 4 },
);
}

View File

@ -10,8 +10,8 @@
// copyright notice, and modified files need to carry a notice indicating
// that they have been altered from the originals.
import { stat } from 'fs/promises';
import { normalize } from 'path';
import { stat } from "fs/promises";
import { normalize } from "path";
export function getRoot() {
return normalize(`${__dirname}/../../`);
@ -22,7 +22,7 @@ export async function pathExists(path: string) {
await stat(path);
return true;
} catch (err: any) {
if (err && err.code === 'ENOENT') return false;
if (err && err.code === "ENOENT") return false;
throw err;
}
}

View File

@ -13,11 +13,11 @@
export type PythonObjectMeta = {
python_api_name?: string;
python_api_type?:
| 'class'
| 'method'
| 'property'
| 'attribute'
| 'module'
| 'function'
| 'exception';
| "class"
| "method"
| "property"
| "attribute"
| "module"
| "function"
| "exception";
};

View File

@ -10,7 +10,7 @@
// copyright notice, and modified files need to carry a notice indicating
// that they have been altered from the originals.
import { PythonObjectMeta } from './PythonObjectMeta';
import { PythonObjectMeta } from "./PythonObjectMeta";
export type SphinxToMdResult = {
markdown: string;

View File

@ -10,8 +10,8 @@
// copyright notice, and modified files need to carry a notice indicating
// that they have been altered from the originals.
import { getLastPartFromFullIdentifier } from '../stringUtils';
import { SphinxToMdResult } from './SphinxToMdResult';
import { getLastPartFromFullIdentifier } from "../stringUtils";
import { SphinxToMdResult } from "./SphinxToMdResult";
export function addFrontMatter<T extends SphinxToMdResult>(results: T[]): T[] {
for (let result of results) {

View File

@ -10,18 +10,18 @@
// copyright notice, and modified files need to carry a notice indicating
// that they have been altered from the originals.
import { describe, expect, test } from '@jest/globals';
import { dedupeIds } from './dedupeIds';
import { describe, expect, test } from "@jest/globals";
import { dedupeIds } from "./dedupeIds";
describe('dedupeIds', () => {
test('dedupeIds', async () => {
describe("dedupeIds", () => {
test("dedupeIds", async () => {
expect(
await dedupeIds(`
<span id="foo" />
<span id="bar" />
# foo
<span id="foo" />
`)
`),
).toMatchInlineSnapshot(`
"<span id="bar" />

View File

@ -10,20 +10,22 @@
// copyright notice, and modified files need to carry a notice indicating
// that they have been altered from the originals.
import { unified } from 'unified';
import remarkParse from 'remark-parse';
import remarkMath from 'remark-math';
import remarkGfm from 'remark-gfm';
import remarkMdx from 'remark-mdx';
import { Root } from 'mdast';
import { visit } from 'unist-util-visit';
import remarkStringify from 'remark-stringify';
import { remarkStringifyOptions } from './unifiedParser';
import { toText } from 'hast-util-to-text';
import Slugger from 'github-slugger';
import { SphinxToMdResult } from './SphinxToMdResult';
import { unified } from "unified";
import remarkParse from "remark-parse";
import remarkMath from "remark-math";
import remarkGfm from "remark-gfm";
import remarkMdx from "remark-mdx";
import { Root } from "mdast";
import { visit } from "unist-util-visit";
import remarkStringify from "remark-stringify";
import { remarkStringifyOptions } from "./unifiedParser";
import { toText } from "hast-util-to-text";
import Slugger from "github-slugger";
import { SphinxToMdResult } from "./SphinxToMdResult";
export async function dedupeResultIds<T extends SphinxToMdResult>(results: T[]): Promise<T[]> {
export async function dedupeResultIds<T extends SphinxToMdResult>(
results: T[],
): Promise<T[]> {
for (let result of results) {
result.markdown = await dedupeIds(result.markdown);
}
@ -41,14 +43,16 @@ export async function dedupeIds(md: string): Promise<string> {
const existingIds = new Set();
const slugger = new Slugger();
visit(tree, 'heading', (node) => {
visit(tree, "heading", (node) => {
const headingText = toText(node as any);
existingIds.add(slugger.slug(headingText));
});
visit(tree, 'mdxJsxFlowElement', (node, index, parent) => {
if (node.name === 'span') {
const id = node.attributes?.find((attr) => 'name' in attr && attr.name === 'id')?.value;
visit(tree, "mdxJsxFlowElement", (node, index, parent) => {
if (node.name === "span") {
const id = node.attributes?.find(
(attr) => "name" in attr && attr.name === "id",
)?.value;
if (id) {
if (existingIds.has(id) && parent !== null && index !== null) {
parent.children.splice(index, 1);

View File

@ -10,10 +10,12 @@
// copyright notice, and modified files need to carry a notice indicating
// that they have been altered from the originals.
import { SphinxToMdResultWithUrl } from './SphinxToMdResult';
import { removePart } from '../stringUtils';
import { SphinxToMdResultWithUrl } from "./SphinxToMdResult";
import { removePart } from "../stringUtils";
export function flatFolders<T extends SphinxToMdResultWithUrl>(results: T[]): T[] {
export function flatFolders<T extends SphinxToMdResultWithUrl>(
results: T[],
): T[] {
for (const result of results) {
result.url = omitRootFolders(result.url);
}
@ -21,5 +23,5 @@ export function flatFolders<T extends SphinxToMdResultWithUrl>(results: T[]): T[
}
function omitRootFolders(path: string): string {
return removePart(path, '/', ['stubs', 'apidocs', 'apidoc']);
return removePart(path, "/", ["stubs", "apidocs", "apidoc"]);
}

View File

@ -10,59 +10,62 @@
// copyright notice, and modified files need to carry a notice indicating
// that they have been altered from the originals.
import { describe, expect, test } from '@jest/globals';
import { generateToc } from './generateToc';
import { describe, expect, test } from "@jest/globals";
import { generateToc } from "./generateToc";
describe('generateTocFromPythonApiFiles', () => {
test('generate a toc', () => {
describe("generateTocFromPythonApiFiles", () => {
test("generate a toc", () => {
const toc = generateToc({
pkg,
results: [
{ meta: {}, url: '/docs/runtime' },
{ meta: {}, url: "/docs/runtime" },
{
meta: {
python_api_type: 'module',
python_api_name: 'qiskit_ibm_runtime',
python_api_type: "module",
python_api_name: "qiskit_ibm_runtime",
},
url: '/docs/runtime/qiskit_ibm_runtime',
url: "/docs/runtime/qiskit_ibm_runtime",
},
{
meta: {
python_api_type: 'module',
python_api_name: 'qiskit_ibm_runtime.options',
python_api_type: "module",
python_api_name: "qiskit_ibm_runtime.options",
},
url: '/docs/runtime/qiskit_ibm_runtime.options',
url: "/docs/runtime/qiskit_ibm_runtime.options",
},
{
meta: { python_api_type: 'class', python_api_name: 'Sampler' },
url: '/docs/runtime/qiskit_ibm_runtime.Sampler',
meta: { python_api_type: "class", python_api_name: "Sampler" },
url: "/docs/runtime/qiskit_ibm_runtime.Sampler",
},
{
meta: { python_api_type: 'method', python_api_name: 'Sampler.run' },
url: '/docs/runtime/qiskit_ibm_runtime.Sampler.run',
meta: { python_api_type: "method", python_api_name: "Sampler.run" },
url: "/docs/runtime/qiskit_ibm_runtime.Sampler.run",
},
{
meta: { python_api_type: 'class', python_api_name: 'Estimator' },
url: '/docs/runtime/qiskit_ibm_runtime.Estimator',
meta: { python_api_type: "class", python_api_name: "Estimator" },
url: "/docs/runtime/qiskit_ibm_runtime.Estimator",
},
{
meta: { python_api_type: 'class' },
url: '/docs/runtime/qiskit_ibm_runtime.NoName',
meta: { python_api_type: "class" },
url: "/docs/runtime/qiskit_ibm_runtime.NoName",
},
{
meta: { python_api_type: 'class', python_api_name: 'Options' },
url: 'qiskit_ibm_runtime.options.Options',
},
{
meta: { python_api_type: 'function', python_api_name: 'runSomething' },
url: 'qiskit_ibm_runtime.runSomething',
meta: { python_api_type: "class", python_api_name: "Options" },
url: "qiskit_ibm_runtime.options.Options",
},
{
meta: {
python_api_type: 'module',
python_api_name: 'qiskit_ibm_runtime.single',
python_api_type: "function",
python_api_name: "runSomething",
},
url: '/docs/runtime/qiskit_ibm_runtime/single',
url: "qiskit_ibm_runtime.runSomething",
},
{
meta: {
python_api_type: "module",
python_api_name: "qiskit_ibm_runtime.single",
},
url: "/docs/runtime/qiskit_ibm_runtime/single",
},
],
});
@ -98,38 +101,44 @@ describe('generateTocFromPythonApiFiles', () => {
`);
});
test('nest modules', () => {
test("nest modules", () => {
const toc = generateToc({
pkg,
results: [
{
meta: {
python_api_type: 'module',
python_api_name: 'qiskit_ibm_runtime',
python_api_type: "module",
python_api_name: "qiskit_ibm_runtime",
},
url: '/docs/runtime/qiskit_ibm_runtime',
url: "/docs/runtime/qiskit_ibm_runtime",
},
{
meta: {
python_api_type: 'module',
python_api_name: 'qiskit_ibm_runtime.options',
python_api_type: "module",
python_api_name: "qiskit_ibm_runtime.options",
},
url: '/docs/runtime/qiskit_ibm_runtime.options',
},
{
meta: { python_api_type: 'class', python_api_name: 'qiskit_ibm_runtime.Estimator' },
url: '/docs/runtime/qiskit_ibm_runtime.Estimator',
},
{
meta: { python_api_type: 'class', python_api_name: 'qiskit_ibm_runtime.options.Options' },
url: 'qiskit_ibm_runtime.options.Options',
url: "/docs/runtime/qiskit_ibm_runtime.options",
},
{
meta: {
python_api_type: 'class',
python_api_name: 'qiskit_ibm_runtime.options.Options2',
python_api_type: "class",
python_api_name: "qiskit_ibm_runtime.Estimator",
},
url: 'qiskit_ibm_runtime.options.Options2',
url: "/docs/runtime/qiskit_ibm_runtime.Estimator",
},
{
meta: {
python_api_type: "class",
python_api_name: "qiskit_ibm_runtime.options.Options",
},
url: "qiskit_ibm_runtime.options.Options",
},
{
meta: {
python_api_type: "class",
python_api_name: "qiskit_ibm_runtime.options.Options2",
},
url: "qiskit_ibm_runtime.options.Options2",
},
],
});
@ -178,13 +187,13 @@ describe('generateTocFromPythonApiFiles', () => {
`);
});
test('skip nest modules using a fn', () => {
test("skip nest modules using a fn", () => {
const toc = generateToc({
pkg: {
...pkg,
tocOptions: {
nestModule(id: string) {
if (id === 'qiskit_ibm_runtime.options') return false;
if (id === "qiskit_ibm_runtime.options") return false;
return true;
},
},
@ -192,46 +201,52 @@ describe('generateTocFromPythonApiFiles', () => {
results: [
{
meta: {
python_api_type: 'module',
python_api_name: 'qiskit_ibm_runtime',
python_api_type: "module",
python_api_name: "qiskit_ibm_runtime",
},
url: '/docs/runtime/qiskit_ibm_runtime',
url: "/docs/runtime/qiskit_ibm_runtime",
},
{
meta: {
python_api_type: 'module',
python_api_name: 'qiskit_ibm_runtime.options',
python_api_type: "module",
python_api_name: "qiskit_ibm_runtime.options",
},
url: '/docs/runtime/qiskit_ibm_runtime.options',
},
{
meta: { python_api_type: 'class', python_api_name: 'qiskit_ibm_runtime.Estimator' },
url: '/docs/runtime/qiskit_ibm_runtime.Estimator',
},
{
meta: { python_api_type: 'class', python_api_name: 'qiskit_ibm_runtime.options.Options' },
url: 'qiskit_ibm_runtime.options.Options',
url: "/docs/runtime/qiskit_ibm_runtime.options",
},
{
meta: {
python_api_type: 'class',
python_api_name: 'qiskit_ibm_runtime.options.Options2',
python_api_type: "class",
python_api_name: "qiskit_ibm_runtime.Estimator",
},
url: 'qiskit_ibm_runtime.options.Options2',
url: "/docs/runtime/qiskit_ibm_runtime.Estimator",
},
{
meta: {
python_api_type: 'module',
python_api_name: 'qiskit_ibm_runtime.provider',
python_api_type: "class",
python_api_name: "qiskit_ibm_runtime.options.Options",
},
url: 'qiskit_ibm_runtime.provider',
url: "qiskit_ibm_runtime.options.Options",
},
{
meta: {
python_api_type: 'class',
python_api_name: 'qiskit_ibm_runtime.provider.Provider',
python_api_type: "class",
python_api_name: "qiskit_ibm_runtime.options.Options2",
},
url: 'qiskit_ibm_runtime.provider.Provider',
url: "qiskit_ibm_runtime.options.Options2",
},
{
meta: {
python_api_type: "module",
python_api_name: "qiskit_ibm_runtime.provider",
},
url: "qiskit_ibm_runtime.provider",
},
{
meta: {
python_api_type: "class",
python_api_name: "qiskit_ibm_runtime.provider.Provider",
},
url: "qiskit_ibm_runtime.provider.Provider",
},
],
});
@ -295,8 +310,8 @@ describe('generateTocFromPythonApiFiles', () => {
});
const pkg = {
title: 'Qiskit Runtime IBM Client',
name: 'qiskit_ibm_runtime',
version: '1.0.0',
title: "Qiskit Runtime IBM Client",
name: "qiskit_ibm_runtime",
version: "1.0.0",
changelogUrl: `https://github.com/qiskit_ibm_runtime/releases`,
};

View File

@ -10,9 +10,9 @@
// copyright notice, and modified files need to carry a notice indicating
// that they have been altered from the originals.
import { isEmpty, keyBy, keys, orderBy } from 'lodash';
import { getLastPartFromFullIdentifier } from '../stringUtils';
import { PythonObjectMeta } from './PythonObjectMeta';
import { isEmpty, keyBy, keys, orderBy } from "lodash";
import { getLastPartFromFullIdentifier } from "../stringUtils";
import { PythonObjectMeta } from "./PythonObjectMeta";
type TocEntry = {
title: string;
@ -42,24 +42,28 @@ export function generateToc(options: {
}) {
const { pkg, results } = options;
const nestModule = options.pkg.tocOptions?.nestModule ?? (() => true);
const resultsWithName = results.filter((result) => !isEmpty(result.meta.python_api_name));
const modules = resultsWithName.filter((result) => result.meta.python_api_type === 'module');
const items = resultsWithName.filter(
(result) =>
result.meta.python_api_type === 'class' ||
result.meta.python_api_type === 'function' ||
result.meta.python_api_type === 'exception'
const resultsWithName = results.filter(
(result) => !isEmpty(result.meta.python_api_name),
);
const tocChildren: Toc['children'] = [];
const modules = resultsWithName.filter(
(result) => result.meta.python_api_type === "module",
);
const items = resultsWithName.filter(
(result) =>
result.meta.python_api_type === "class" ||
result.meta.python_api_type === "function" ||
result.meta.python_api_type === "exception",
);
const tocChildren: Toc["children"] = [];
if (modules.length > 0) {
const tocModules = modules.map(
(module): TocEntry => ({
title: module.meta.python_api_name!,
url: module.url,
})
}),
);
const tocModulesByTitle = keyBy(tocModules, (toc) => toc.title);
const tocModuleTitles = keys(tocModulesByTitle);
@ -67,8 +71,13 @@ export function generateToc(options: {
// Add items to modules
for (const item of items) {
if (!item.meta.python_api_name) continue;
const itemModuleTitle = findClosestParentModules(item.meta.python_api_name, tocModuleTitles);
const itemModule = itemModuleTitle ? tocModulesByTitle[itemModuleTitle] : undefined;
const itemModuleTitle = findClosestParentModules(
item.meta.python_api_name,
tocModuleTitles,
);
const itemModule = itemModuleTitle
? tocModulesByTitle[itemModuleTitle]
: undefined;
if (itemModule) {
if (!itemModule.children) itemModule.children = [];
const itemTocEntry: TocEntry = {
@ -87,8 +96,13 @@ export function generateToc(options: {
continue;
}
const parentModuleTitle = findClosestParentModules(tocModule.title, tocModuleTitles);
const parentModule = parentModuleTitle ? tocModulesByTitle[parentModuleTitle] : undefined;
const parentModuleTitle = findClosestParentModules(
tocModule.title,
tocModuleTitles,
);
const parentModule = parentModuleTitle
? tocModulesByTitle[parentModuleTitle]
: undefined;
if (parentModule) {
if (!parentModule.children) parentModule.children = [];
parentModule.children.push(tocModule);
@ -101,7 +115,7 @@ export function generateToc(options: {
for (const tocModule of tocModules) {
if (tocModule.children && tocModule.children.length > 0) {
tocModule.children = [
{ title: 'Overview', url: tocModule.url },
{ title: "Overview", url: tocModule.url },
...orderEntriesByChildrenAndTitle(tocModule.children),
];
delete tocModule.url;
@ -112,11 +126,15 @@ export function generateToc(options: {
}
tocChildren.push({
title: 'Changelog',
title: "Changelog",
url: pkg.changelogUrl,
});
const toc: Toc = { title: pkg.title, subtitle: `v${pkg.version}`, children: tocChildren };
const toc: Toc = {
title: pkg.title,
subtitle: `v${pkg.version}`,
children: tocChildren,
};
if (pkg.tocOptions?.collapsed) {
toc.collapsed = true;
}
@ -124,9 +142,9 @@ export function generateToc(options: {
}
function findClosestParentModules(id: string, possibleParents: string[]) {
const idParts = id.split('.');
const idParts = id.split(".");
for (let i = idParts.length - 1; i > 0; i--) {
const testId = idParts.slice(0, i).join('.');
const testId = idParts.slice(0, i).join(".");
if (possibleParents.includes(testId)) {
return testId;
}

View File

@ -10,11 +10,11 @@
// copyright notice, and modified files need to carry a notice indicating
// that they have been altered from the originals.
import { describe, expect, test } from '@jest/globals';
import { mergeClassMembers } from './mergeClassMembers';
import { describe, expect, test } from "@jest/globals";
import { mergeClassMembers } from "./mergeClassMembers";
describe('mergeClassMembers', () => {
test('merge class members', async () => {
describe("mergeClassMembers", () => {
test("merge class members", async () => {
const results: Parameters<typeof mergeClassMembers>[0] = [
{
markdown: `## Attributes
@ -29,10 +29,10 @@ describe('mergeClassMembers', () => {
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------- |
| [\`RuntimeOptions.validate\`](qiskit_ibm_runtime.RuntimeOptions.validate#qiskit_ibm_runtime.RuntimeOptions.validate "qiskit_ibm_runtime.RuntimeOptions.validate")(channel) | Validate options. |`,
meta: {
python_api_type: 'class',
python_api_name: 'RuntimeOptions',
python_api_type: "class",
python_api_name: "RuntimeOptions",
},
url: '/docs/api/qiskit-ibm-runtime/stubs/qiskit_ibm_runtime.RuntimeOptions',
url: "/docs/api/qiskit-ibm-runtime/stubs/qiskit_ibm_runtime.RuntimeOptions",
images: [],
},
{
@ -41,10 +41,10 @@ describe('mergeClassMembers', () => {
\`Optional[str] = None\`
`,
meta: {
python_api_type: 'attribute',
python_api_name: 'RuntimeOptions.backend',
python_api_type: "attribute",
python_api_name: "RuntimeOptions.backend",
},
url: '/docs/api/qiskit-ibm-runtime/stubs/qiskit_ibm_runtime.RuntimeOptions.backend',
url: "/docs/api/qiskit-ibm-runtime/stubs/qiskit_ibm_runtime.RuntimeOptions.backend",
images: [],
},
{
@ -53,10 +53,10 @@ describe('mergeClassMembers', () => {
\`Optional[str] = None\`
`,
meta: {
python_api_type: 'property',
python_api_name: 'RuntimeOptions.circuits',
python_api_type: "property",
python_api_name: "RuntimeOptions.circuits",
},
url: '/docs/api/qiskit-ibm-runtime/stubs/qiskit_ibm_runtime.RuntimeOptions.circuits',
url: "/docs/api/qiskit-ibm-runtime/stubs/qiskit_ibm_runtime.RuntimeOptions.circuits",
images: [],
},
{
@ -80,16 +80,17 @@ Validate options.
\`None\`
`,
meta: {
python_api_type: 'method',
python_api_name: 'RuntimeOptions.backend',
python_api_type: "method",
python_api_name: "RuntimeOptions.backend",
},
url: '/docs/api/qiskit-ibm-runtime/stubs/qiskit_ibm_runtime.RuntimeOptions.validate',
url: "/docs/api/qiskit-ibm-runtime/stubs/qiskit_ibm_runtime.RuntimeOptions.validate",
images: [],
},
];
const merged = await mergeClassMembers(results);
expect(merged.find((item) => item.meta.python_api_type === 'class')?.markdown)
.toMatchInlineSnapshot(`
expect(
merged.find((item) => item.meta.python_api_type === "class")?.markdown,
).toMatchInlineSnapshot(`
"## Attributes
### RuntimeOptions.backend

View File

@ -10,39 +10,53 @@
// copyright notice, and modified files need to carry a notice indicating
// that they have been altered from the originals.
import { includes, isEmpty, orderBy, reject } from 'lodash';
import { unified } from 'unified';
import remarkParse from 'remark-parse';
import remarkMdx from 'remark-mdx';
import remarkGfm from 'remark-gfm';
import remarkMath from 'remark-math';
import remarkStringify from 'remark-stringify';
import { Content, Root } from 'mdast';
import { visit } from 'unist-util-visit';
import { SphinxToMdResultWithUrl } from './SphinxToMdResult';
import { remarkStringifyOptions } from './unifiedParser';
import { includes, isEmpty, orderBy, reject } from "lodash";
import { unified } from "unified";
import remarkParse from "remark-parse";
import remarkMdx from "remark-mdx";
import remarkGfm from "remark-gfm";
import remarkMath from "remark-math";
import remarkStringify from "remark-stringify";
import { Content, Root } from "mdast";
import { visit } from "unist-util-visit";
import { SphinxToMdResultWithUrl } from "./SphinxToMdResult";
import { remarkStringifyOptions } from "./unifiedParser";
export async function mergeClassMembers<T extends SphinxToMdResultWithUrl>(
results: T[]
results: T[],
): Promise<T[]> {
const resultsWithName = results.filter((result) => !isEmpty(result.meta.python_api_name));
const classes = resultsWithName.filter((result) => result.meta.python_api_type === 'class');
const resultsWithName = results.filter(
(result) => !isEmpty(result.meta.python_api_name),
);
const classes = resultsWithName.filter(
(result) => result.meta.python_api_type === "class",
);
for (const clazz of classes) {
const members = orderBy(
resultsWithName.filter((result) => {
if (!includes(['method', 'property', 'attribute', 'function'], result.meta.python_api_type))
if (
!includes(
["method", "property", "attribute", "function"],
result.meta.python_api_type,
)
)
return false;
return result.meta.python_api_name?.startsWith(`${clazz.meta.python_api_name}.`);
return result.meta.python_api_name?.startsWith(
`${clazz.meta.python_api_name}.`,
);
}),
(result) => result.meta.python_api_name
(result) => result.meta.python_api_name,
);
const attributesAndProps = members.filter(
(member) =>
member.meta.python_api_type === 'attribute' || member.meta.python_api_type === 'property'
member.meta.python_api_type === "attribute" ||
member.meta.python_api_type === "property",
);
const methods = members.filter(
(member) => member.meta.python_api_type === "method",
);
const methods = members.filter((member) => member.meta.python_api_type === 'method');
try {
// inject members markdown
@ -55,9 +69,19 @@ export async function mergeClassMembers<T extends SphinxToMdResultWithUrl>(
.use(() => {
return async (tree) => {
for (const node of tree.children) {
await replaceMembersAfterTitle(tree, node, 'Attributes', attributesAndProps);
await replaceMembersAfterTitle(tree, node, 'Methods', methods);
await replaceMembersAfterTitle(tree, node, 'Methods Defined Here', methods);
await replaceMembersAfterTitle(
tree,
node,
"Attributes",
attributesAndProps,
);
await replaceMembersAfterTitle(tree, node, "Methods", methods);
await replaceMembersAfterTitle(
tree,
node,
"Methods Defined Here",
methods,
);
}
};
})
@ -65,7 +89,7 @@ export async function mergeClassMembers<T extends SphinxToMdResultWithUrl>(
.process(clazz.markdown)
).toString();
} catch (e) {
console.log('Error found in', clazz.meta.python_api_name);
console.log("Error found in", clazz.meta.python_api_name);
console.log(clazz.markdown);
throw e;
}
@ -73,7 +97,7 @@ export async function mergeClassMembers<T extends SphinxToMdResultWithUrl>(
// remove merged results
const finalResults = reject(results, (result) =>
includes(['method', 'attribute', 'property'], result.meta.python_api_type)
includes(["method", "attribute", "property"], result.meta.python_api_type),
);
return finalResults;
@ -83,16 +107,20 @@ async function replaceMembersAfterTitle(
tree: Root,
node: Content,
title: string,
members: SphinxToMdResultWithUrl[]
members: SphinxToMdResultWithUrl[],
) {
if (node.type !== 'heading') return;
if (node.type !== "heading") return;
const nodeIndex = tree.children.indexOf(node);
if (nodeIndex === -1) return;
const nextNode = tree.children[nodeIndex + 1];
const firstChild = node.children[0];
if (firstChild?.type === 'text' && firstChild?.value === title && nextNode?.type === 'table') {
if (
firstChild?.type === "text" &&
firstChild?.value === title &&
nextNode?.type === "table"
) {
const children: any[] = [];
for (const member of members) {
const updated = await parseMarkdownIncreasingHeading(member.markdown, 2);
@ -102,7 +130,10 @@ async function replaceMembersAfterTitle(
}
}
async function parseMarkdownIncreasingHeading(md: string, depthIncrease: number): Promise<Root> {
async function parseMarkdownIncreasingHeading(
md: string,
depthIncrease: number,
): Promise<Root> {
const root = await unified()
.use(remarkParse)
.use(remarkGfm)
@ -114,7 +145,7 @@ async function parseMarkdownIncreasingHeading(md: string, depthIncrease: number)
.use(remarkGfm)
.use(remarkMdx)
.use(() => (root) => {
visit(root, 'heading', (node: any) => {
visit(root, "heading", (node: any) => {
node.depth = node.depth + depthIncrease;
});
})

View File

@ -10,11 +10,11 @@
// copyright notice, and modified files need to carry a notice indicating
// that they have been altered from the originals.
import { describe, test, expect } from '@jest/globals';
import { sphinxHtmlToMarkdown } from './sphinxHtmlToMarkdown';
import { describe, test, expect } from "@jest/globals";
import { sphinxHtmlToMarkdown } from "./sphinxHtmlToMarkdown";
describe('sphinxHtmlToMarkdown', () => {
test('remove .html extension from relative links', async () => {
describe("sphinxHtmlToMarkdown", () => {
test("remove .html extension from relative links", async () => {
expect(
await toMd(`<div
role='main'
@ -54,7 +54,7 @@ describe('sphinxHtmlToMarkdown', () => {
</div>
</section>
</article>
</div>`)
</div>`),
).toMatchInlineSnapshot(`
"<span id="qiskit-ibm-runtime-api-reference" />
@ -66,7 +66,7 @@ describe('sphinxHtmlToMarkdown', () => {
`);
});
test('remove permalink', async () => {
test("remove permalink", async () => {
expect(
await toMd(`<article role="main">
<section id="qiskit-ibm-runtime-api-reference">
@ -78,7 +78,7 @@ describe('sphinxHtmlToMarkdown', () => {
</ul>
</div>
</section>
</article>`)
</article>`),
).toMatchInlineSnapshot(`
"<span id="qiskit-ibm-runtime-api-reference" />
@ -90,7 +90,7 @@ describe('sphinxHtmlToMarkdown', () => {
`);
});
test('remove download links', async () => {
test("remove download links", async () => {
expect(
await toMd(`<div
role='main'
@ -100,11 +100,11 @@ describe('sphinxHtmlToMarkdown', () => {
>
<p>(<a class="reference download internal" download="" href="../_downloads/366189d70d6a05b2c91f442d20ba6114/qiskit-circuit-QuantumCircuit-1.py"><code class="xref download docutils literal notranslate"><span class="pre">Source</span> <span class="pre">code</span></code></a>)</p>
</div>
`)
`),
).toMatchInlineSnapshot(`""`);
});
test('extract images', async () => {
test("extract images", async () => {
expect(
await sphinxHtmlToMarkdown({
html: `
@ -113,9 +113,9 @@ describe('sphinxHtmlToMarkdown', () => {
<img src="http://google.com/bar.png"/>
</div>
`,
url: 'http://qiskit.org/docs/quantum-circuit.html',
imageDestination: '/images/qiskit',
})
url: "http://qiskit.org/docs/quantum-circuit.html",
imageDestination: "/images/qiskit",
}),
).toMatchInlineSnapshot(`
{
"images": [
@ -135,7 +135,7 @@ describe('sphinxHtmlToMarkdown', () => {
`);
});
test('handle tabs', async () => {
test("handle tabs", async () => {
expect(
await toMd(`<div role='main'>
<details
@ -219,7 +219,7 @@ describe('sphinxHtmlToMarkdown', () => {
</div>
</details>
</div>
`)
`),
).toMatchInlineSnapshot(`
"### Account initialization
@ -230,7 +230,7 @@ describe('sphinxHtmlToMarkdown', () => {
`);
});
test('handle tables', async () => {
test("handle tables", async () => {
expect(
await toMd(`
<div role='main'>
@ -415,7 +415,7 @@ describe('sphinxHtmlToMarkdown', () => {
</tbody>
</table>
</div>
`)
`),
).toMatchInlineSnapshot(`
"| | |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------- |
@ -434,7 +434,7 @@ describe('sphinxHtmlToMarkdown', () => {
`);
});
test('handle <', async () => {
test("handle <", async () => {
expect(
await toMd(`
<div role='main'>
@ -448,7 +448,7 @@ describe('sphinxHtmlToMarkdown', () => {
</dl>
</div>
`)
`),
).toMatchInlineSnapshot(`
"For the full list of backend attributes, see the IBMBackend class documentation \\<[https://qiskit.org/documentation/apidoc/providers\\_models.html](https://qiskit.org/documentation/apidoc/providers_models.html)>
@ -459,7 +459,7 @@ describe('sphinxHtmlToMarkdown', () => {
`);
});
test('handle {', async () => {
test("handle {", async () => {
expect(
await toMd(`
<div role='main'>
@ -467,14 +467,14 @@ describe('sphinxHtmlToMarkdown', () => {
Can be either (1) a dictionary mapping XX angle values to fidelity at that angle; or
(2) a single float f, interpreted as {pi: f, pi/2: f/2, pi/3: f/3}.</p>
</div>
`)
`),
).toMatchInlineSnapshot(`
"**basis\\_fidelity** (*dict | float*) available strengths and fidelity of each. Can be either (1) a dictionary mapping XX angle values to fidelity at that angle; or (2) a single float f, interpreted as \\{pi: f, pi/2: f/2, pi/3: f/3}.
"
`);
});
test('translate codeblocks to code fences with lang python', async () => {
test("translate codeblocks to code fences with lang python", async () => {
expect(
await toMd(`
<div role='main'>
@ -484,7 +484,7 @@ Can be either (1) a dictionary mapping XX angle values to fidelity at that angle
<span class='n'>filters</span><span class='o'>=</span><span class='k'>lambda</span> <span class='n'>x</span><span class='p'>:</span> <span class='p'>(</span><span class='s2'>&quot;rz&quot;</span> <span class='ow'>in</span> <span class='n'>x</span><span class='o'>.</span><span class='n'>basis_gates</span> <span class='p'>)</span>
</pre>
</div>
`)
`),
).toMatchInlineSnapshot(`
"\`\`\`python
QiskitRuntimeService.backends(
@ -496,7 +496,7 @@ Can be either (1) a dictionary mapping XX angle values to fidelity at that angle
`);
});
test('convert source links', async () => {
test("convert source links", async () => {
expect(
(
await sphinxHtmlToMarkdown({
@ -504,17 +504,17 @@ Can be either (1) a dictionary mapping XX angle values to fidelity at that angle
<span class='sig-prename descclassname'><span class='pre'>IBMBackend.</span></span><span class='sig-name descname'><span class='pre'>control_channel</span></span><span class='sig-paren'>(</span><em class='sig-param'><span class='n'><span class='pre'>qubits</span></span></em><span class='sig-paren'>)</span><a class='reference internal' href='../_modules/qiskit_ibm_runtime/ibm_backend.html#IBMBackend.control_channel'><span class='viewcode-link'><span class='pre'>[source]</span></span></a><a class='headerlink' href='#qiskit_ibm_runtime.IBMBackend.control_channel' title='Permalink to this definition'></a>
</div>
`,
url: 'https://qiskit.org/documentation/partners/qiskit_ibm_runtime/stubs/qiskit_ibm_runtime.Sampler.html',
url: "https://qiskit.org/documentation/partners/qiskit_ibm_runtime/stubs/qiskit_ibm_runtime.Sampler.html",
baseSourceUrl: `https://github.com/Qiskit/qiskit-ibm-runtime/tree/0.9.2/`,
})
).markdown
).markdown,
).toMatchInlineSnapshot(`
"IBMBackend.control\\_channel(*qubits*)[\\[source\\]](https://github.com/Qiskit/qiskit-ibm-runtime/tree/0.9.2/qiskit_ibm_runtime/ibm_backend.py)
"
`);
});
test('convert class signature headings', async () => {
test("convert class signature headings", async () => {
expect(
await toMdWithMeta(`<div role='main'>
<h1>Estimator<a class='headerlink' href='#estimator' title='Permalink to this heading'></a></h1>
@ -572,7 +572,7 @@ Can be either (1) a dictionary mapping XX angle values to fidelity at that angle
</dd>
</dl>
</div>
`)
`),
).toMatchInlineSnapshot(`
{
"images": [],
@ -592,7 +592,7 @@ Can be either (1) a dictionary mapping XX angle values to fidelity at that angle
`);
});
test('convert class property headings', async () => {
test("convert class property headings", async () => {
expect(
await toMdWithMeta(`<div role='main'>
<h1>Estimator.circuits<a class='headerlink' href='#estimator' title='Permalink to this heading'></a></h1>
@ -622,7 +622,7 @@ Can be either (1) a dictionary mapping XX angle values to fidelity at that angle
<dd><p>Quantum circuits that represents quantum states.</p></dd>
</dl>
</div>
`)
`),
).toMatchInlineSnapshot(`
{
"images": [],
@ -642,7 +642,7 @@ Can be either (1) a dictionary mapping XX angle values to fidelity at that angle
`);
});
test('convert class method headings', async () => {
test("convert class method headings", async () => {
expect(
await toMdWithMeta(`<div role='main'>
<h1>Estimator.run<a class='headerlink' href='#estimator' title='Permalink to this heading'></a></h1>
@ -651,7 +651,7 @@ Can be either (1) a dictionary mapping XX angle values to fidelity at that angle
<span class='sig-prename descclassname'><span class='pre'>Estimator.</span></span><span class='sig-name descname'><span class='pre'>run</span></span><span class='sig-paren'>(</span><em class='sig-param'><span class='n'><span class='pre'>circuits</span></span></em>, <em class='sig-param'><span class='n'><span class='pre'>observables</span></span></em>, <em class='sig-param'><span class='n'><span class='pre'>parameter_values</span></span><span class='o'><span class='pre'>=</span></span><span class='default_value'><span class='pre'>None</span></span></em>, <em class='sig-param'><span class='o'><span class='pre'>**</span></span><span class='n'><span class='pre'>kwargs</span></span></em><span class='sig-paren'>)</span><a class='reference internal' href='../_modules/qiskit_ibm_runtime/estimator.html#Estimator.run'><span class='viewcode-link'><span class='pre'>[source]</span></span></a><a class='headerlink' href='#qiskit_ibm_runtime.Estimator.run' title='Permalink to this definition'></a></dt>
<dd><p>Submit a request to the estimator primitive program.</p></dd></dl>
</div>
`)
`),
).toMatchInlineSnapshot(`
{
"images": [],
@ -671,7 +671,7 @@ Can be either (1) a dictionary mapping XX angle values to fidelity at that angle
`);
});
test('convert class attributes headings', async () => {
test("convert class attributes headings", async () => {
expect(
await toMdWithMeta(`<div role='main'>
<h1>EnvironmentOptions.callback<a class='headerlink' href='#estimator' title='Permalink to this heading'></a></h1>
@ -680,7 +680,7 @@ Can be either (1) a dictionary mapping XX angle values to fidelity at that angle
<span class='sig-prename descclassname'><span class='pre'>EnvironmentOptions.</span></span><span class='sig-name descname'><span class='pre'>callback</span></span><em class='property'><span class='p'><span class='pre'>:</span></span><span class='w'> </span><span class='pre'>Optional</span><span class='p'><span class='pre'>[</span></span><span class='pre'>Callable</span><span class='p'><span class='pre'>]</span></span></em><em class='property'><span class='w'> </span><span class='p'><span class='pre'>=</span></span><span class='w'> </span><span class='pre'>None</span></em><a class='headerlink' href='#qiskit_ibm_runtime.options.EnvironmentOptions.callback' title='Permalink to this definition'></a></dt>
<dd></dd></dl>
</div>
`)
`),
).toMatchInlineSnapshot(`
{
"images": [],
@ -698,23 +698,23 @@ Can be either (1) a dictionary mapping XX angle values to fidelity at that angle
`);
});
test('convert method and attributes to titles', async () => {
test("convert method and attributes to titles", async () => {
expect(
(
await toMdWithMeta(
`<div role='main'>
<p class='rubric'>Methods</p>
</div>
`
`,
)
).markdown
).markdown,
).toMatchInlineSnapshot(`
"## Methods
"
`);
});
test('convert functions headings', async () => {
test("convert functions headings", async () => {
expect(
await toMdWithMeta(`<div role='main'>
<section id="job-monitor">
@ -740,7 +740,7 @@ By default this is sys.stdout.</p></li>
</section>
</div>
`)
`),
).toMatchInlineSnapshot(`
{
"images": [],
@ -772,7 +772,7 @@ By default this is sys.stdout.</p></li>
`);
});
test('convert exception headings', async () => {
test("convert exception headings", async () => {
expect(
await toMdWithMeta(`<div role='main'>
<article itemprop="articleBody" id="pytorch-article" class="pytorch-article">
@ -791,7 +791,7 @@ By default this is sys.stdout.</p></li>
</article>
</div>
`)
`),
).toMatchInlineSnapshot(`
{
"images": [],
@ -815,7 +815,7 @@ By default this is sys.stdout.</p></li>
`);
});
test('extract module meta for .target', async () => {
test("extract module meta for .target", async () => {
expect(
(
await toMdWithMeta(
@ -823,9 +823,9 @@ By default this is sys.stdout.</p></li>
<article itemprop="articleBody" id="pytorch-article" class="pytorch-article">
<span class="target" id="module-qiskit_ibm_runtime"></span>
</article>
</div>`
</div>`,
)
).meta
).meta,
).toMatchInlineSnapshot(`
{
"python_api_name": "qiskit_ibm_runtime",
@ -834,7 +834,7 @@ By default this is sys.stdout.</p></li>
`);
});
test('extract module meta for section', async () => {
test("extract module meta for section", async () => {
expect(
await toMdWithMeta(`<div role="main"><section id="module-qiskit_ibm_provider.transpiler.passes.basis">
<span id="basis"></span><h1>basis<a class="headerlink" href="#module-qiskit_ibm_provider.transpiler.passes.basis" title="Permalink to this heading"></a></h1>
@ -842,7 +842,7 @@ By default this is sys.stdout.</p></li>
<h2>Basis (<a class="reference internal" href="#module-qiskit_ibm_provider.transpiler.passes.basis" title="qiskit_ibm_provider.transpiler.passes.basis"><code class="xref py py-mod docutils literal notranslate"><span class="pre">qiskit_ibm_provider.transpiler.passes.basis</span></code></a>)<a class="headerlink" href="#basis-qiskit-ibm-provider-transpiler-passes-basis" title="Permalink to this heading"></a></h2>
<p>Passes to layout circuits to IBM backends instruction sets.</p>
</section>
</section></div>`)
</section></div>`),
).toMatchInlineSnapshot(`
{
"images": [],
@ -870,7 +870,7 @@ By default this is sys.stdout.</p></li>
`);
});
test('convert class method with a problematic output', async () => {
test("convert class method with a problematic output", async () => {
// The problem is generated in this page
// https://qiskit.org/ecosystem/ibm-provider/stubs/qiskit_ibm_provider.job.IBMCircuitJob.wait_for_final_state.html#qiskit_ibm_provider.job.IBMCircuitJob.wait_for_final_state
expect(
@ -918,7 +918,7 @@ By default this is sys.stdout.</p></li>
</dd>
</dl>
</div>
`)
`),
).toMatchInlineSnapshot(`
{
"images": [],
@ -952,7 +952,7 @@ By default this is sys.stdout.</p></li>
`);
});
test('convert inline methods', async () => {
test("convert inline methods", async () => {
expect(
await toMd(`
<div role="main">
@ -990,7 +990,7 @@ bits.</p>
</dd></dl>
</div>
`)
`),
).toMatchInlineSnapshot(`
"# DAGCircuit
@ -1032,7 +1032,7 @@ bits.</p>
`);
});
test('transform dl, dd, dt elements', async () => {
test("transform dl, dd, dt elements", async () => {
expect(
await toMd(`<div role='main'>
<dl>
@ -1049,7 +1049,7 @@ bits.</p>
</dd>
</dl>
</div>
`)
`),
).toMatchInlineSnapshot(`
"## Return type
@ -1062,14 +1062,14 @@ bits.</p>
`);
});
test('remove () around module titles', async () => {
test("remove () around module titles", async () => {
expect(
await toMd(`<div role='main'>
<span class="target" id="module-qiskit_ibm_runtime"></span><section id="qiskit-runtime-qiskit-ibm-runtime">
<h1>Qiskit Runtime (<a class="reference internal" href="#module-qiskit_ibm_runtime" title="qiskit_ibm_runtime"><code class="xref py py-mod docutils literal notranslate"><span class="pre">qiskit_ibm_runtime</span></code></a>)<a class="headerlink" href="#qiskit-runtime-qiskit-ibm-runtime" title="Permalink to this heading"></a></h1>
<p>Modules related to Qiskit Runtime IBM Client.</p>
</div>
`)
`),
).toMatchInlineSnapshot(`
"<span id="module-qiskit_ibm_runtime" />
@ -1086,7 +1086,7 @@ bits.</p>
`);
});
test('transform inline math', async () => {
test("transform inline math", async () => {
expect(
await toMd(`
<div role='main'>
@ -1098,7 +1098,7 @@ bits.</p>
</tbody>
</table>
</div>
`)
`),
).toMatchInlineSnapshot(`
"| | |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- |
@ -1107,7 +1107,7 @@ bits.</p>
`);
});
test('transform block math', async () => {
test("transform block math", async () => {
expect(
await toMd(`
<div role='main'>
@ -1128,7 +1128,7 @@ bits.</p>
<div class="math notranslate nohighlight">
\\[x = \\sum_{i=0}^{n-1} 2^i q_i,\\]</div>
</div>
`)
`),
).toMatchInlineSnapshot(`
"$$
\\begin{split}CCX q_0, q_1, q_2 =
@ -1152,7 +1152,7 @@ bits.</p>
`);
});
test('transform admonitions', async () => {
test("transform admonitions", async () => {
expect(
await toMd(`<div role='main'>
<div class='admonition note'>
@ -1171,7 +1171,7 @@ bits.</p>
<p>The global phase gate (<span class="math notranslate nohighlight">\\(e^{i\\theta}\\)</span>).</p>
</div>
</div>
`)
`),
).toMatchInlineSnapshot(`
"<Admonition title="Note" type="note">
To use these tools locally, youll need to install the additional dependencies for the visualization functions:
@ -1188,7 +1188,7 @@ bits.</p>
`);
});
test('parse inline attributes section', async () => {
test("parse inline attributes section", async () => {
expect(
await toMd(`<div role='main'>
@ -1230,7 +1230,7 @@ bits.</p>
<dd>Bar has a type and a defualt value</dd>
</dl>
</div>
`)
`),
).toMatchInlineSnapshot(`
"<span id="quantumcircuit" />
@ -1281,7 +1281,7 @@ bits.</p>
`);
});
test('parse deprecations warnings', async () => {
test("parse deprecations warnings", async () => {
expect(
await toMd(`
<div role="main">
@ -1289,7 +1289,7 @@ bits.</p>
<p><span class="versionmodified deprecated">Deprecated since version 0.23.0: </span>The method <code class="docutils literal notranslate"><span class="pre">qiskit.circuit.quantumregister.QuantumRegister.qasm()</span></code> is deprecated as of qiskit-terra 0.23.0. It will be removed no earlier than 3 months after the release date. Correct exporting to OpenQASM 2 is the responsibility of a larger exporter; it cannot safely be done on an object-by-object basis without context. No replacement will be provided, because the premise is wrong.</p>
</div>
</div>
`)
`),
).toMatchInlineSnapshot(`
"<Admonition title="Deprecated since version 0.23.0" type="danger">
The method \`qiskit.circuit.quantumregister.QuantumRegister.qasm()\` is deprecated as of qiskit-terra 0.23.0. It will be removed no earlier than 3 months after the release date. Correct exporting to OpenQASM 2 is the responsibility of a larger exporter; it cannot safely be done on an object-by-object basis without context. No replacement will be provided, because the premise is wrong.
@ -1298,7 +1298,7 @@ bits.</p>
`);
});
test('preserve span with ids', async () => {
test("preserve span with ids", async () => {
expect(
await toMd(`<div role='main'>
<div role='main' class='main-content' itemscope='itemscope' itemtype='http://schema.org/Article'>
@ -1318,7 +1318,7 @@ bits.</p>
</section>
</article>
</div>
`)
`),
).toMatchInlineSnapshot(`
"<span id="module-qiskit.assembler" />
@ -1339,7 +1339,7 @@ bits.</p>
`);
});
test('merge contiguous emphasis', async () => {
test("merge contiguous emphasis", async () => {
expect(
await toMd(`
<div role="main">
@ -1347,14 +1347,14 @@ bits.</p>
<li><p><strong>gate</strong> (<em>Union</em><em>[</em><a class="reference internal" href="qiskit.circuit.Gate.html#qiskit.circuit.Gate" title="qiskit.circuit.Gate"><em>Gate</em></a><em>, </em><em>str</em><em>]</em>) Gate information.</p></li>
</div>
`)
`),
).toMatchInlineSnapshot(`
"* **gate** (*Union\\[*[*Gate*](qiskit.circuit.Gate#qiskit.circuit.Gate "qiskit.circuit.Gate")*, str]*) Gate information.
"
`);
});
test('remove spaces from emphashis boundaries', async () => {
test("remove spaces from emphashis boundaries", async () => {
expect(
await toMd(`
<div role="main">
@ -1362,14 +1362,14 @@ bits.</p>
<li><p><strong>gate</strong> (<em> Union</em><em>[</em><a class="reference internal" href="qiskit.circuit.Gate.html#qiskit.circuit.Gate" title="qiskit.circuit.Gate"><em>Gate</em></a><em>, </em><em>str</em><em>] </em>) Gate information.</p></li>
</div>
`)
`),
).toMatchInlineSnapshot(`
"* **gate** ( *Union\\[*[*Gate*](qiskit.circuit.Gate#qiskit.circuit.Gate "qiskit.circuit.Gate")*, str]* ) Gate information.
"
`);
});
test('remove <br/>', async () => {
test("remove <br/>", async () => {
expect(
await toMd(`
<div role="main">
@ -1387,7 +1387,7 @@ compilation flow follows the structure given below:</p>
<br><p>Qiskit has four pre-built transpilation pipelines available here:
</p>
</div>
`)
`),
).toMatchInlineSnapshot(`
"Transpilation is the process of rewriting a given input circuit to match the topology of a specific quantum device, and/or to optimize the circuit for execution on present day noisy quantum systems.
@ -1404,7 +1404,7 @@ compilation flow follows the structure given below:</p>
async function toMd(html: string) {
return (
await sphinxHtmlToMarkdown({
url: 'https://qiskit.org/documentation/partners/qiskit_ibm_runtime/stubs/qiskit_ibm_runtime.Sampler.html',
url: "https://qiskit.org/documentation/partners/qiskit_ibm_runtime/stubs/qiskit_ibm_runtime.Sampler.html",
html,
})
).markdown;
@ -1412,7 +1412,7 @@ async function toMd(html: string) {
async function toMdWithMeta(html: string) {
return await sphinxHtmlToMarkdown({
url: 'https://qiskit.org/documentation/partners/qiskit_ibm_runtime/stubs/qiskit_ibm_runtime.Sampler.html',
url: "https://qiskit.org/documentation/partners/qiskit_ibm_runtime/stubs/qiskit_ibm_runtime.Sampler.html",
html,
});
}

View File

@ -10,24 +10,28 @@
// copyright notice, and modified files need to carry a notice indicating
// that they have been altered from the originals.
import { load } from 'cheerio';
import { unified } from 'unified';
import rehypeParse from 'rehype-parse';
import rehypeRemark from 'rehype-remark';
import remarkStringify from 'remark-stringify';
import remarkGfm from 'remark-gfm';
import { last, first, without, initial, tail } from 'lodash';
import { defaultHandlers, Handle, toMdast, all } from 'hast-util-to-mdast';
import { toText } from 'hast-util-to-text';
import remarkMath from 'remark-math';
import remarkMdx from 'remark-mdx';
import { SphinxToMdResult } from './SphinxToMdResult';
import { PythonObjectMeta } from './PythonObjectMeta';
import { getLastPartFromFullIdentifier, removePrefix, removeSuffix } from '../stringUtils';
import { remarkStringifyOptions } from './unifiedParser';
import { MdxJsxFlowElement } from 'mdast-util-mdx-jsx';
import { visit } from 'unist-util-visit';
import { Root } from 'mdast';
import { load } from "cheerio";
import { unified } from "unified";
import rehypeParse from "rehype-parse";
import rehypeRemark from "rehype-remark";
import remarkStringify from "remark-stringify";
import remarkGfm from "remark-gfm";
import { last, first, without, initial, tail } from "lodash";
import { defaultHandlers, Handle, toMdast, all } from "hast-util-to-mdast";
import { toText } from "hast-util-to-text";
import remarkMath from "remark-math";
import remarkMdx from "remark-mdx";
import { SphinxToMdResult } from "./SphinxToMdResult";
import { PythonObjectMeta } from "./PythonObjectMeta";
import {
getLastPartFromFullIdentifier,
removePrefix,
removeSuffix,
} from "../stringUtils";
import { remarkStringifyOptions } from "./unifiedParser";
import { MdxJsxFlowElement } from "mdast-util-mdx-jsx";
import { visit } from "unist-util-visit";
import { Root } from "mdast";
export async function sphinxHtmlToMarkdown(options: {
html: string;
@ -38,7 +42,12 @@ export async function sphinxHtmlToMarkdown(options: {
baseSourceUrl?: string;
}): Promise<SphinxToMdResult> {
const images: Array<{ src: string; dest: string }> = [];
const { html, url, imageDestination = '/images/api/', baseSourceUrl } = options;
const {
html,
url,
imageDestination = "/images/api/",
baseSourceUrl,
} = options;
const meta: PythonObjectMeta = {};
const $page = load(html);
@ -46,27 +55,27 @@ export async function sphinxHtmlToMarkdown(options: {
const $main = $page(main);
// remove html extensions in relative links
$main.find('a').each((_, link) => {
$main.find("a").each((_, link) => {
const $link = $page(link);
const href = $link.attr('href');
if (href && !href.startsWith('http')) {
$link.attr('href', href.replaceAll('.html', ''));
const href = $link.attr("href");
if (href && !href.startsWith("http")) {
$link.attr("href", href.replaceAll(".html", ""));
}
});
$main
.find('img')
.find("img")
.toArray()
.forEach((el) => {
const $img = $page(el);
const imageUrl = new URL($img.attr('src')!, url);
const imageUrl = new URL($img.attr("src")!, url);
const src = imageUrl.toString();
const filename = last(src.split('/'));
const filename = last(src.split("/"));
const dest = `${imageDestination}/${filename}`;
$img.attr('src', dest);
$img.attr("src", dest);
images.push({ src, dest: dest });
});
@ -77,62 +86,66 @@ export async function sphinxHtmlToMarkdown(options: {
$main.find('a[title="Link to this definition"]').remove();
// remove download source code
$main.find('p > a.reference.download.internal').closest('p').remove();
$main.find("p > a.reference.download.internal").closest("p").remove();
// handle tabs, use heading for the summary and remove the blockquote
$main.find('.sd-summary-title').each((_, quote) => {
$main.find(".sd-summary-title").each((_, quote) => {
const $quote = $page(quote);
$quote.replaceWith(`<h3>${$quote.html()}</h3>`);
});
$main.find('.sd-card-body blockquote').each((_, quote) => {
$main.find(".sd-card-body blockquote").each((_, quote) => {
const $quote = $page(quote);
$quote.replaceWith($quote.children());
});
// add language class to code blocks
$main.find('pre').each((_, pre) => {
$main.find("pre").each((_, pre) => {
const $pre = $page(pre);
$pre.replaceWith(`<pre><code class="language-python">${$pre.html()}</code></pre>`);
$pre.replaceWith(
`<pre><code class="language-python">${$pre.html()}</code></pre>`,
);
});
// replace source links
$main.find('a').each((_, a) => {
$main.find("a").each((_, a) => {
const $a = $page(a);
const href = $a.attr('href');
if (href?.startsWith('http:')) return;
const href = $a.attr("href");
if (href?.startsWith("http:")) return;
if (href?.includes(`/_modules/`)) {
//_modules/qiskit_ibm_runtime/ibm_backend
const match = href?.match(/_modules\/(.*?)(#|$)/);
if (match) {
const newHref = `${baseSourceUrl ?? ''}${match[1]}.py`;
$a.attr('href', newHref);
const newHref = `${baseSourceUrl ?? ""}${match[1]}.py`;
$a.attr("href", newHref);
}
}
});
// use titles for method and attribute headers
$main.find('.rubric').each((_, el) => {
$main.find(".rubric").each((_, el) => {
const $el = $page(el);
$el.replaceWith(`<h2>${$el.html()}</h2>`);
});
// delete colons
$main.find('.colon').remove();
$main.find(".colon").remove();
// translate type headings to titles
function findByText(selector: string, text: string) {
return $main.find(selector).filter((i, el) => $page(el).text().trim() === text);
return $main
.find(selector)
.filter((i, el) => $page(el).text().trim() === text);
}
$main
.find('dl.field-list.simple')
.find("dl.field-list.simple")
.toArray()
.map((dl) => {
const $dl = $page(dl);
$dl
.find('dt')
.find("dt")
.toArray()
.forEach((dt) => {
const $dt = $page(dt);
@ -140,7 +153,7 @@ export async function sphinxHtmlToMarkdown(options: {
});
$dl
.find('dd')
.find("dd")
.toArray()
.forEach((dd) => {
const $dd = $page(dd);
@ -155,7 +168,7 @@ export async function sphinxHtmlToMarkdown(options: {
// members can be recursive, so we need to pick elements one by one
const dl = $main
.find(
'dl.py.class, dl.py.property, dl.py.method, dl.py.attribute, dl.py.function, dl.py.exception'
"dl.py.class, dl.py.property, dl.py.method, dl.py.attribute, dl.py.function, dl.py.exception",
)
.get(0);
@ -170,65 +183,67 @@ export async function sphinxHtmlToMarkdown(options: {
.toArray()
.map((child) => {
const $child = $page(child);
$child.find('.viewcode-link').closest('a').remove();
const id = $dl.find('dt.sig-object').attr('id');
$child.find(".viewcode-link").closest("a").remove();
const id = $dl.find("dt.sig-object").attr("id");
if (child.name === 'dt' && $dl.hasClass('class')) {
if (child.name === "dt" && $dl.hasClass("class")) {
if (!meta.python_api_type) {
meta.python_api_type = 'class';
meta.python_api_type = "class";
meta.python_api_name = id;
}
findByText('em.property', 'class').remove();
findByText("em.property", "class").remove();
return `<span class="target" id="${id}"/><p><code>${$child.html()}</code></p>`;
} else if (child.name === 'dt' && $dl.hasClass('property')) {
} else if (child.name === "dt" && $dl.hasClass("property")) {
if (!meta.python_api_type) {
meta.python_api_type = 'property';
meta.python_api_type = "property";
meta.python_api_name = id;
if (id) {
$dl.siblings('h1').text(getLastPartFromFullIdentifier(id));
$dl.siblings("h1").text(getLastPartFromFullIdentifier(id));
}
}
findByText('em.property', 'property').remove();
const signature = $child.find('em').text()?.replace(/^:\s+/, '');
findByText("em.property", "property").remove();
const signature = $child.find("em").text()?.replace(/^:\s+/, "");
if (signature.trim().length === 0) return;
return `<span class="target" id='${id}'/><p><code>${signature}</code></p>`;
} else if (child.name === 'dt' && $dl.hasClass('method')) {
} else if (child.name === "dt" && $dl.hasClass("method")) {
if (!meta.python_api_type) {
meta.python_api_type = 'method';
meta.python_api_type = "method";
meta.python_api_name = id;
if (id) {
$dl.siblings('h1').text(getLastPartFromFullIdentifier(id));
$dl.siblings("h1").text(getLastPartFromFullIdentifier(id));
}
} else {
// Inline methods
if (id) {
$page(`<h3>${getLastPartFromFullIdentifier(id)}</h3>`).insertBefore($dl);
$page(
`<h3>${getLastPartFromFullIdentifier(id)}</h3>`,
).insertBefore($dl);
}
}
findByText('em.property', 'method').remove();
findByText("em.property", "method").remove();
return `<span class="target" id='${id}'/><p><code>${$child.html()}</code></p>`;
} else if (child.name === 'dt' && $dl.hasClass('attribute')) {
} else if (child.name === "dt" && $dl.hasClass("attribute")) {
if (!meta.python_api_type) {
meta.python_api_type = 'attribute';
meta.python_api_type = "attribute";
meta.python_api_name = id;
if (id) {
$dl.siblings('h1').text(getLastPartFromFullIdentifier(id));
$dl.siblings("h1").text(getLastPartFromFullIdentifier(id));
}
findByText('em.property', 'attribute').remove();
const signature = $child.find('em').text()?.replace(/^:\s+/, '');
findByText("em.property", "attribute").remove();
const signature = $child.find("em").text()?.replace(/^:\s+/, "");
if (signature.trim().length === 0) return;
return `<span class="target" id='${id}'/><p><code>${signature}</code></p>`;
} else {
// The attribute is embedded on the class
const text = $child.text();
const equalIndex = text.indexOf('=');
const colonIndex = text.indexOf(':');
const equalIndex = text.indexOf("=");
const colonIndex = text.indexOf(":");
let name = text;
let type: string | undefined;
let value: string | undefined;
@ -243,42 +258,44 @@ export async function sphinxHtmlToMarkdown(options: {
name = text.substring(0, equalIndex);
value = text.substring(equalIndex);
}
const output = [`<span class="target" id='${id}'/><h3>${name}</h3>`];
const output = [
`<span class="target" id='${id}'/><h3>${name}</h3>`,
];
if (type) {
output.push(`<p><code>${type}</code></p>`);
}
if (value) {
output.push(`<p><code>${value}</code></p>`);
}
return output.join('\n');
return output.join("\n");
}
} else if (child.name === 'dt' && $dl.hasClass('function')) {
} else if (child.name === "dt" && $dl.hasClass("function")) {
if (!meta.python_api_type) {
meta.python_api_type = 'function';
meta.python_api_type = "function";
meta.python_api_name = id;
}
findByText('em.property', 'function').remove();
findByText("em.property", "function").remove();
return `<span class="target" id="${id}"/><p><code>${$child.html()}</code></p>`;
} else if (child.name === 'dt' && $dl.hasClass('exception')) {
} else if (child.name === "dt" && $dl.hasClass("exception")) {
if (!meta.python_api_type) {
meta.python_api_type = 'exception';
meta.python_api_type = "exception";
meta.python_api_name = id;
}
findByText('em.property', 'exception').remove();
findByText("em.property", "exception").remove();
return `<span class="target" id='${id}'/><p><code>${$child.html()}</code></p>`;
}
return `<div>${$child.html()}</div>`;
})
.join('\n');
.join("\n");
$dl.replaceWith(`<div>${replacement}</div>`);
}
// preserve math block whitespace
$main
.find('div.math')
.find("div.math")
.toArray()
.map((el) => {
const $el = $page(el);
@ -286,31 +303,31 @@ export async function sphinxHtmlToMarkdown(options: {
});
// extract module meta
const modulePrefix = 'module-';
const modulePrefix = "module-";
const moduleIdWithPrefix = $main
.find(`.target, section`)
.toArray()
.map((el) => $page(el).attr('id'))
.map((el) => $page(el).attr("id"))
.find((id) => id?.startsWith(modulePrefix));
if (moduleIdWithPrefix) {
const moduleId = moduleIdWithPrefix.slice(modulePrefix.length);
meta.python_api_type = 'module';
meta.python_api_type = "module";
meta.python_api_name = moduleId;
}
// Update headings of modules
if (meta.python_api_type === 'module') {
if (meta.python_api_type === "module") {
$main
.find('h1,h2')
.find("h1,h2")
.toArray()
.forEach((el) => {
const $el = $page(el);
const $a = $page($el.find('a'));
const $a = $page($el.find("a"));
const signature = $a.text();
$a.remove();
let title = $el.text();
title = title.replace('()', '');
title = title.replace("()", "");
let replacement = `<${el.tagName}>${title}</${el.tagName}>`;
if (signature.trim().length > 0) {
replacement += `<span class="target" id="module-${meta.python_api_name}" /><p><code>${signature}</code></p>`;
@ -333,17 +350,17 @@ export async function sphinxHtmlToMarkdown(options: {
return all(h, node);
},
span(h, node: any) {
if (node.properties.className?.includes('math')) {
if (node.properties.className?.includes("math")) {
let value = node.children[0].value;
const prefix = '\\(';
const sufix = '\\)';
const prefix = "\\(";
const sufix = "\\)";
if (value.startsWith(prefix) && value.endsWith(sufix)) {
value = value.substring(prefix.length, value.length - sufix.length);
}
return { type: 'inlineMath', value };
return { type: "inlineMath", value };
}
if (node.properties.id && node.properties.className?.includes('target')) {
if (node.properties.id && node.properties.className?.includes("target")) {
return [buildSpanId(node.properties.id), ...all(h, node)];
}
@ -354,14 +371,14 @@ export async function sphinxHtmlToMarkdown(options: {
return all(h, node);
},
pre(h, node: any) {
if (node.properties.className?.includes('math')) {
if (node.properties.className?.includes("math")) {
let value = node.children[0].value;
const prefix = '\\[';
const sufix = '\\]';
const prefix = "\\[";
const sufix = "\\]";
if (value.startsWith(prefix) && value.endsWith(sufix)) {
value = value.substring(prefix.length, value.length - sufix.length);
}
return { type: 'math', value };
return { type: "math", value };
}
return defaultHandlers.pre(h, node);
},
@ -372,17 +389,20 @@ export async function sphinxHtmlToMarkdown(options: {
return defaultHandlers.div(h, node);
},
dt(h, node: any) {
if (meta.python_api_type === 'class' || meta.python_api_type === 'module') {
if (
meta.python_api_type === "class" ||
meta.python_api_type === "module"
) {
return [
h(node, 'strong', {
type: 'strong',
h(node, "strong", {
type: "strong",
children: all(h, node),
}),
{ type: 'text', value: ' ' },
{ type: "text", value: " " },
];
}
return h(node, 'heading', {
type: 'heading',
return h(node, "heading", {
type: "heading",
depth: 2,
children: all(h, node),
});
@ -390,41 +410,47 @@ export async function sphinxHtmlToMarkdown(options: {
div(h, node: any): any {
const nodeClasses = node.properties.className ?? [];
if (nodeClasses.includes('admonition')) {
const titleNode = node.children.find((child: any) =>
child.properties.className?.includes('admonition-title')
if (nodeClasses.includes("admonition")) {
const titleNode = node.children.find(
(child: any) =>
child.properties.className?.includes("admonition-title"),
);
let type = 'note';
if (nodeClasses.includes('warning')) {
type = 'caution';
} else if (nodeClasses.includes('important')) {
type = 'danger';
let type = "note";
if (nodeClasses.includes("warning")) {
type = "caution";
} else if (nodeClasses.includes("important")) {
type = "danger";
}
const otherChildren = without(node.children, titleNode);
return buildAdmonition({
title: toText(titleNode),
type,
children: otherChildren.map((node: any) => toMdast(node, { handlers })),
children: otherChildren.map((node: any) =>
toMdast(node, { handlers }),
),
});
} else if (nodeClasses.includes('deprecated')) {
} else if (nodeClasses.includes("deprecated")) {
const root = node.children[0];
const titleNode = root.children.find((child: any) =>
child.properties.className?.includes('versionmodified')
const titleNode = root.children.find(
(child: any) =>
child.properties.className?.includes("versionmodified"),
);
let title = toText(titleNode).trim();
if (title.endsWith(':')) {
if (title.endsWith(":")) {
title = title.slice(0, -1);
}
const otherChildren = without(root.children, titleNode);
return buildAdmonition({
title,
type: 'danger',
type: "danger",
children: [
{
type: 'paragraph',
children: otherChildren.map((node: any) => toMdast(node, { handlers })),
type: "paragraph",
children: otherChildren.map((node: any) =>
toMdast(node, { handlers }),
),
},
],
});
@ -446,21 +472,23 @@ export async function sphinxHtmlToMarkdown(options: {
.use(() => {
return (root: Root) => {
// merge contiguous emphasis
visit(root, 'emphasis', (node, index, parent) => {
visit(root, "emphasis", (node, index, parent) => {
if (index === null || parent === null) return;
let nextIndex = index + 1;
while (parent.children[nextIndex]?.type === 'emphasis') {
node.children.push(...((parent.children[nextIndex] as any).children ?? []));
while (parent.children[nextIndex]?.type === "emphasis") {
node.children.push(
...((parent.children[nextIndex] as any).children ?? []),
);
nextIndex++;
}
parent.children.splice(index + 1, nextIndex - (index + 1));
});
// remove initial and trailing spaces from emphasis
visit(root, 'emphasis', (node, index, parent) => {
visit(root, "emphasis", (node, index, parent) => {
if (index === null || parent === null) return;
const firstChild = first(node.children);
if (firstChild?.type === 'text') {
if (firstChild?.type === "text") {
const match = firstChild.value.match(/^\s+/);
if (match) {
if (match[0] === firstChild.value) {
@ -468,11 +496,14 @@ export async function sphinxHtmlToMarkdown(options: {
} else {
firstChild.value = removePrefix(firstChild.value, match[0]);
}
parent.children.splice(index, 0, { type: 'text', value: match[0] });
parent.children.splice(index, 0, {
type: "text",
value: match[0],
});
}
}
const lastChild = last(node.children);
if (lastChild?.type === 'text') {
if (lastChild?.type === "text") {
const match = lastChild.value.match(/\s+$/);
if (match) {
if (match[0] === lastChild.value) {
@ -480,7 +511,10 @@ export async function sphinxHtmlToMarkdown(options: {
} else {
lastChild.value = removeSuffix(lastChild.value, match[0]);
}
parent.children.splice(index + 1, 0, { type: 'text', value: match[0] });
parent.children.splice(index + 1, 0, {
type: "text",
value: match[0],
});
}
}
});
@ -489,7 +523,7 @@ export async function sphinxHtmlToMarkdown(options: {
.process(mainHtml);
let markdown = mdFile.toString();
markdown = markdown.replaceAll(`<!---->`, '');
markdown = markdown.replaceAll(`<!---->`, "");
return { markdown, meta, images };
}
@ -501,17 +535,17 @@ function buildAdmonition(options: {
}): MdxJsxFlowElement {
const { title, type, children } = options;
return {
type: 'mdxJsxFlowElement',
name: 'Admonition',
type: "mdxJsxFlowElement",
name: "Admonition",
attributes: [
{
type: 'mdxJsxAttribute',
name: 'title',
type: "mdxJsxAttribute",
name: "title",
value: title,
},
{
type: 'mdxJsxAttribute',
name: 'type',
type: "mdxJsxAttribute",
name: "type",
value: type,
},
],
@ -521,12 +555,12 @@ function buildAdmonition(options: {
function buildSpanId(id: string): MdxJsxFlowElement {
return {
type: 'mdxJsxFlowElement',
name: 'span',
type: "mdxJsxFlowElement",
name: "span",
attributes: [
{
type: 'mdxJsxAttribute',
name: 'id',
type: "mdxJsxAttribute",
name: "id",
value: id,
},
],

View File

@ -10,13 +10,13 @@
// copyright notice, and modified files need to carry a notice indicating
// that they have been altered from the originals.
import { describe, expect, test } from '@jest/globals';
import { updateLinks } from './updateLinks';
import { SphinxToMdResultWithUrl } from './SphinxToMdResult';
import { last } from 'lodash';
import { describe, expect, test } from "@jest/globals";
import { updateLinks } from "./updateLinks";
import { SphinxToMdResultWithUrl } from "./SphinxToMdResult";
import { last } from "lodash";
describe('updateLinks', () => {
test('update links', async () => {
describe("updateLinks", () => {
test("update links", async () => {
const input: SphinxToMdResultWithUrl[] = [
{
markdown: `
@ -29,10 +29,10 @@ describe('updateLinks', () => {
[link7](#qiskit_ibm_runtime.RuntimeJob.job)
`,
meta: {
python_api_type: 'class',
python_api_name: 'qiskit_ibm_runtime.RuntimeJob',
python_api_type: "class",
python_api_name: "qiskit_ibm_runtime.RuntimeJob",
},
url: '/docs/api/qiskit-ibm-runtime/stubs/qiskit_ibm_runtime.RuntimeJob',
url: "/docs/api/qiskit-ibm-runtime/stubs/qiskit_ibm_runtime.RuntimeJob",
images: [],
},
{
@ -40,10 +40,10 @@ describe('updateLinks', () => {
[run](qiskit_ibm_runtime.RuntimeJob#qiskit_ibm_runtime.RuntimeJob.run)
`,
meta: {
python_api_type: 'class',
python_api_name: 'qiskit_ibm_runtime.Sampler',
python_api_type: "class",
python_api_name: "qiskit_ibm_runtime.Sampler",
},
url: '/docs/api/qiskit-ibm-runtime/stubs/qiskit_ibm_runtime.RuntimeJob',
url: "/docs/api/qiskit-ibm-runtime/stubs/qiskit_ibm_runtime.RuntimeJob",
images: [],
},
];
@ -81,7 +81,7 @@ describe('updateLinks', () => {
`);
});
test('update links using a transform function', async () => {
test("update links using a transform function", async () => {
const input: SphinxToMdResultWithUrl[] = [
{
markdown: `
@ -92,25 +92,25 @@ describe('updateLinks', () => {
[link7](#qiskit_ibm_runtime.RuntimeJob.job)
`,
meta: {
python_api_type: 'class',
python_api_name: 'qiskit_ibm_runtime.RuntimeJob',
python_api_type: "class",
python_api_name: "qiskit_ibm_runtime.RuntimeJob",
},
url: '/docs/api/qiskit-ibm-runtime/stubs/qiskit_ibm_runtime.RuntimeJob',
url: "/docs/api/qiskit-ibm-runtime/stubs/qiskit_ibm_runtime.RuntimeJob",
images: [],
},
];
const results = await updateLinks(input, (url) => {
let path = last(url.split('/'))!;
if (path.includes('#')) {
path = path.split('#').join('.html#');
let path = last(url.split("/"))!;
if (path.includes("#")) {
path = path.split("#").join(".html#");
} else {
path += '.html';
path += ".html";
}
if (path?.startsWith('algorithms'))
if (path?.startsWith("algorithms"))
return { url: `http://qiskit.org/documentation/apidoc/${path}` };
if (path?.startsWith('qiskit.algorithms.'))
if (path?.startsWith("qiskit.algorithms."))
return { url: `http://qiskit.org/documentation/stubs/${path}` };
});
expect(results).toMatchInlineSnapshot(`

View File

@ -10,25 +10,31 @@
// copyright notice, and modified files need to carry a notice indicating
// that they have been altered from the originals.
import { initial, keyBy, keys, last } from 'lodash';
import { Root } from 'mdast';
import { visit } from 'unist-util-visit';
import isAbsoluteUrl from 'is-absolute-url';
import { removePart, removePrefix } from '../stringUtils';
import { SphinxToMdResultWithUrl } from './SphinxToMdResult';
import { remarkStringifyOptions } from './unifiedParser';
import { unified } from 'unified';
import remarkParse from 'remark-parse';
import remarkMath from 'remark-math';
import remarkGfm from 'remark-gfm';
import remarkMdx from 'remark-mdx';
import remarkStringify from 'remark-stringify';
import { initial, keyBy, keys, last } from "lodash";
import { Root } from "mdast";
import { visit } from "unist-util-visit";
import isAbsoluteUrl from "is-absolute-url";
import { removePart, removePrefix } from "../stringUtils";
import { SphinxToMdResultWithUrl } from "./SphinxToMdResult";
import { remarkStringifyOptions } from "./unifiedParser";
import { unified } from "unified";
import remarkParse from "remark-parse";
import remarkMath from "remark-math";
import remarkGfm from "remark-gfm";
import remarkMdx from "remark-mdx";
import remarkStringify from "remark-stringify";
export async function updateLinks<T extends SphinxToMdResultWithUrl>(
results: T[],
transformLink?: (url: string, text?: string) => { url: string; text?: string } | undefined
transformLink?: (
url: string,
text?: string,
) => { url: string; text?: string } | undefined,
): Promise<T[]> {
const resultsByName = keyBy(results, (result) => result.meta.python_api_name!);
const resultsByName = keyBy(
results,
(result) => result.meta.python_api_name!,
);
const itemNames = new Set(keys(resultsByName));
for (const result of results) {
@ -39,9 +45,12 @@ export async function updateLinks<T extends SphinxToMdResultWithUrl>(
.use(remarkMdx)
.use(() => {
return async (tree: Root) => {
visit(tree, 'link', (node) => {
visit(tree, "link", (node) => {
if (transformLink) {
const textNode = node.children?.[0]?.type === 'text' ? node.children?.[0] : undefined;
const textNode =
node.children?.[0]?.type === "text"
? node.children?.[0]
: undefined;
const transformedLink = transformLink(node.url, textNode?.value);
if (transformedLink) {
node.url = transformedLink.url;
@ -53,37 +62,46 @@ export async function updateLinks<T extends SphinxToMdResultWithUrl>(
}
if (isAbsoluteUrl(node.url)) return;
if (node.url.startsWith('/')) return;
if (node.url.startsWith("/")) return;
node.url = removePart(node.url, '/', ['stubs', 'apidocs', 'apidoc', '..']);
node.url = removePart(node.url, "/", [
"stubs",
"apidocs",
"apidoc",
"..",
]);
const urlParts = node.url.split('/');
const urlParts = node.url.split("/");
const initialUrlParts = initial(urlParts);
const [path, hash] = last(urlParts)!.split('#') as [string, string | undefined];
const [path, hash] = last(urlParts)!.split("#") as [
string,
string | undefined,
];
// qiskit_ibm_runtime.RuntimeJob
// qiskit_ibm_runtime.RuntimeJob#qiskit_ibm_runtime.RuntimeJob
if (itemNames.has(path)) {
if (hash === path) {
node.url = [...initialUrlParts, path].join('/');
node.url = [...initialUrlParts, path].join("/");
return;
}
// qiskit_ibm_runtime.RuntimeJob#qiskit_ibm_runtime.RuntimeJob.job -> qiskit_ibm_runtime.RuntimeJob#job
if (hash?.startsWith(`${path}.`)) {
const member = removePrefix(hash, `${path}.`);
node.url = [...initialUrlParts, path].join('/') + `#${member}`;
node.url = [...initialUrlParts, path].join("/") + `#${member}`;
return;
}
}
// qiskit_ibm_runtime.QiskitRuntimeService.job -> qiskit_ibm_runtime.QiskitRuntimeService#job
const pathParts = path.split('.');
const pathParts = path.split(".");
const member = last(pathParts);
const initialPathParts = initial(pathParts);
const parentName = initialPathParts.join('.');
if ('class' === resultsByName[parentName]?.meta.python_api_type) {
node.url = [...initialUrlParts, parentName].join('/') + '#' + member;
const parentName = initialPathParts.join(".");
if ("class" === resultsByName[parentName]?.meta.python_api_type) {
node.url =
[...initialUrlParts, parentName].join("/") + "#" + member;
}
});
};

View File

@ -10,7 +10,7 @@
// copyright notice, and modified files need to carry a notice indicating
// that they have been altered from the originals.
import { last, split } from 'lodash';
import { last, split } from "lodash";
export function removePart(text: string, separator: string, matcher: string[]) {
return text
@ -34,5 +34,5 @@ export function removeSuffix(text: string, suffix: string) {
}
export function getLastPartFromFullIdentifier(fullIdentifierName: string) {
return last(split(fullIdentifierName, '.'))!;
return last(split(fullIdentifierName, "."))!;
}

View File

@ -10,7 +10,7 @@
// copyright notice, and modified files need to carry a notice indicating
// that they have been altered from the originals.
import { ProcessOutput } from 'zx';
import { ProcessOutput } from "zx";
export function zxMain(mainFn: () => Promise<void>) {
enableCliColors();
@ -23,9 +23,9 @@ export function zxMain(mainFn: () => Promise<void>) {
}
export function enableCliColors() {
process.env.FORCE_COLOR = '3';
process.env.FORCE_COLOR = "3";
}
export function disableCliColors() {
process.env.FORCE_COLOR = '0';
process.env.FORCE_COLOR = "0";
}

View File

@ -1,15 +1,15 @@
{
"compilerOptions": {
"target": "es2021",
"module": "ESNext",
"allowJs": false,
"strict": true,
"noEmit": true,
"esModuleInterop": true,
"moduleResolution": "node",
"incremental": true,
"skipLibCheck": true
},
"include": ["**/*.ts"],
"exclude": ["node_modules"]
}
"compilerOptions": {
"target": "es2021",
"module": "ESNext",
"allowJs": false,
"strict": true,
"noEmit": true,
"esModuleInterop": true,
"moduleResolution": "node",
"incremental": true,
"skipLibCheck": true
},
"include": ["**/*.ts"],
"exclude": ["node_modules"]
}