Commit Graph

36 Commits

Author SHA1 Message Date
Proloy Mishra 8c22b6bcac
fix typo in comment 2022-06-28 19:59:09 +05:30
Grisha Vartanyan 38e0ae590c Reduce max hash in raw strings from u16 to u8 2022-03-23 22:13:55 +01:00
Grisha 4e3dbb3c19
Add test for >65535 hashes in lexing raw string 2022-03-16 06:37:41 +01:00
Nicholas Nethercote 37d9ea745b Improve `scan_escape`.
`scan_escape` currently has a fast path (for when the first char isn't
'\\') and a slow path.

This commit changes `scan_escape` so it only handles the slow path, i.e.
the actual escaping code. The fast path is inlined into the two call
sites.

This change makes the code faster, because there is no function call
overhead on the fast path. (`scan_escape` is a big function and doesn't
get inlined.)

This change also improves readability, because it removes a bunch of
mode checks on the the fast paths.
2022-02-24 17:01:01 +11:00
bors 2a9e0831d6 Auto merge of #91393 - Julian-Wollersberger:lexer_optimization, r=petrochenkov
Optimize `rustc_lexer`

The `cursor.first()` method in `rustc_lexer` now calls the `chars.next()` method instead of `chars.nth_char(0)`.

This allows LLVM to optimize the code better. The biggest win is that `eat_while()` is now fully inlined and generates better assembly. This improves the lexer's performance by 35% in a micro-benchmark I made (Lexing all 18MB of code in the compiler directory). But lexing is only a small part of the overall compilation time, so I don't know how significant it is.

Big thanks to criterion and `cargo asm`.
2021-12-03 13:20:14 +00:00
Julian Wollersberger 1f147a2ed7 Replace `nth_char(0)` with `next()` in `cursor.first()`
and optimize the iterator returned by `tokenize().

This improves lexer performance by 35%
2021-12-01 19:14:10 +01:00
Esteban Kuber 38979a3ba1 udpate comment to be more accurate 2021-11-23 20:37:23 +00:00
Esteban Kuber 5a68abb094 Tokenize emoji as if they were valid indentifiers
In the lexer, consider emojis to be valid identifiers and reject
them later to avoid knock down parse errors.
2021-11-23 20:35:07 +00:00
Matthias Krüger 0a5640b55f use matches!() macro in more places 2021-11-06 16:13:14 +01:00
Matthias Krüger 4457014398 Revert "Auto merge of #89709 - clemenswasser:apply_clippy_suggestions_2, r=petrochenkov"
The PR had some unforseen perf regressions that are not as easy to find.
Revert the PR for now.

This reverts commit 6ae8912a3e, reversing
changes made to 86d6d2b738.
2021-10-15 11:28:23 +02:00
Clemens Wasser 71dd0b928b Apply clippy suggestions 2021-10-10 15:38:19 +02:00
Mark Rousskov c746be2219 Migrate to 2021 2021-09-20 22:21:42 -04:00
Frank Steffahn 2396fad095 Fix more “a”/“an” typos 2021-08-22 17:27:18 +02:00
Anton Golov 07aacf53c5 Renamed variable str -> tail for clarity 2021-08-11 13:57:28 +02:00
Anton Golov a03fbfe2ff Warn when an escaped newline skips multiple lines 2021-08-11 11:35:08 +02:00
bors b53a93db2d Auto merge of #87535 - lf-:authors, r=Mark-Simulacrum
rfc3052 followup: Remove authors field from Cargo manifests

Since RFC 3052 soft deprecated the authors field, hiding it from
crates.io, docs.rs, and making Cargo not add it by default, and it is
not generally up to date/useful information for contributors, we may as well
remove it from crates in this repo.
2021-08-02 05:49:17 +00:00
Anton Golov 5d59b4412e Add warning when whitespace is not skipped after an escaped newline. 2021-07-30 16:26:39 +02:00
Jade 3cf820e17d rfc3052: Remove authors field from Cargo manifests
Since RFC 3052 soft deprecated the authors field anyway, hiding it from
crates.io, docs.rs, and making Cargo not add it by default, and it is
not generally up to date/useful information, we should remove it from
crates in this repo.
2021-07-29 14:56:05 -07:00
Ibraheem Ahmed a397fdcc38
Remove ASCII fast path from rustc_lexer::{is_id_continue, is_id_start} 2021-07-26 20:17:28 -04:00
Mara Bos 0eeeebc990 Rename 'bad prefix' to 'unknown prefix'. 2021-06-26 23:11:14 +08:00
Mara Bos 6adce70a58 Improve comments for reserved prefixes.
Co-authored-by: Niko Matsakis <niko@alum.mit.edu>
2021-06-26 23:11:13 +08:00
lrh2000 8dee9bc8fc Reserve prefixed identifiers and string literals (RFC 3101)
This commit denies any identifiers immediately followed by
one of three tokens `"`, `'` or `#`, which is stricter than
the requirements of RFC 3101 but may be necessary according
to the discussion at [Zulip].

[Zulip]: https://rust-lang.zulipchat.com/#narrow/stream/268952-edition-2021/topic/reserved.20prefixes/near/238470099
2021-06-26 23:09:43 +08:00
pierwill 0019ca9141 Fix outdated crate names in compiler docs
Changes `librustc_X` to `rustc_X`, only in documentation comments.
Plain code comments are left unchanged.

Also fix incorrect file paths.
2021-04-08 11:12:14 -05:00
Hanzhen Liang f942c3cbf4 Return EOF_CHAR constant instead of magic char. 2021-01-07 13:20:04 +01:00
Hirochika Matsumoto 56530a2f25 Fix typo 2020-12-18 22:13:25 +09:00
Joshua Nelson 0ad3dce83a Fix some clippy lints 2020-12-03 17:08:19 -05:00
Joshua Nelson 5339bd1ebe Add back missing comments 2020-10-30 10:13:41 -04:00
Joshua Nelson 57c6ed0c07 Fix even more clippy warnings 2020-10-30 10:13:39 -04:00
Julian Wollersberger bd49ded308 Noticed a potential bug in `eat_while()`: it doesn't account for number of UTF8 bytes.
Fixed it by inlining it in the two places where the count is used and simplified the logic there.
2020-10-09 11:12:54 +02:00
LingMan fc20b7841c
Fix typo in rustc_lexer docs
Also add an Oxford comma while we're editing that line.
2020-09-21 05:43:39 +02:00
Vadim Petrochenkov b1491eacfc lexer: Tiny improvement to shebang detection
Lexer now discerns between regular comments and doc comments, so use that.
The change only affects the choice of reported errors.
2020-09-02 00:40:19 +03:00
Aleksey Kladov 5716c3e18d Update expect-test to 1.0
The only change is that `expect_file` now uses path relative to the
current file (same as `include!`). Before, it used paths relative to
the workspace root, which makes no sense.
2020-08-31 21:04:09 +02:00
Aleksey Kladov 7d2460e5c4
Rollup merge of #76050 - matklad:pos, r=petrochenkov
Remove unused function
2020-08-31 15:22:40 +02:00
Vadim Petrochenkov 7dfbf59f10 cleanup: Remove duplicate library names from `Cargo.toml`s 2020-08-30 22:57:54 +03:00
Aleksey Kladov ccffea5b6b Move lexer unit tests to rustc_lexer
StringReader is an intornal abstraction which at the moment changes a
lot, so these unit tests cause quite a bit of friction.

Moving them to rustc_lexer and more ingerated-testing style should
make them much less annoying, hopefully without decreasing their
usefulness much.

Note that coloncolon tests are removed (it's unclear what those are
testing).

\r\n tests are removed as well, as we normalize line endings even
before lexing.
2020-08-30 19:53:36 +02:00
mark 9e5f7d5631 mv compiler to compiler/ 2020-08-30 18:45:07 +03:00