Merge #96

96: [WIP] Begin to add some contributing docs (#95) r=matklad a=DJMcNab

Currently adds some documentation about `cargo gen-kinds`.

Note that I am unsure of some of the details, such as why `DOT` is in `multi_byte_tokens`, hence the [WIP] tag.

Based on #95.

Co-authored-by: Daniel McNab <36049421+djmcnab@users.noreply.github.com>

.cargo/config

@@ -1,4 +1,5 @@
 [alias]
+# Automatically generates the ast and syntax kinds files
 gen-kinds =    "run --package tools -- gen-kinds"
 gen-tests =    "run --package tools -- gen-tests"
 install-code = "run --package tools -- install-code"

CONTRIBUTING.md

@@ -1,38 +1,58 @@
-The project is in its early stages: contributions are welcome and
-would be **very** helpful, but the project is not *yet* optimized for
-contribution. Moreover, it is doubly experimental, so there's no
-guarantee that any work here would reach production. That said, here
-are some areas where contributions would be **especially** welcome:
+The project is in its early stages: contributions are welcome and would be
+**very** helpful, but the project is not _yet_ optimized for contribution.
+Moreover, it is doubly experimental, so there's no guarantee that any work here
+would reach production. That said, here are some areas where contributions would
+be **especially** welcome:
 
+- Designing internal data structures: RFC only outlines the constraints, it's an
+  open question how to satisfy them in the optimal way. See `ARCHITECTURE.md`
+  for current design questions.
 
-* Designing internal data structures: RFC only outlines the
-  constraints, it's an open question how to satisfy them in the
-  optimal way. See `ARCHITECTURE.md` for current design questions.
+- Porting libsyntax parser to rust-analyzer: currently rust-analyzer parses only
+  a tiny subset of Rust. This should be fixed by porting parsing functions from
+  libsyntax one by one. Take a look at the [libsyntax parser] for "what to port"
+  and at the [Kotlin parser] for "how to port".
 
-* Porting libsyntax parser to rust-analyzer: currently rust-analyzer parses
-  only a tiny subset of Rust. This should be fixed by porting parsing
-  functions from libsyntax one by one. Take a look at the
-  [libsyntax parser](https://github.com/rust-lang/rust/blob/6b99adeb11313197f409b4f7c4083c2ceca8a4fe/src/libsyntax/parse/parser.rs)
-  for "what to port" and at the
-  [Kotlin parser](https://github.com/JetBrains/kotlin/blob/4d951de616b20feca92f3e9cc9679b2de9e65195/compiler/frontend/src/org/jetbrains/kotlin/parsing/KotlinParsing.java)
-  for "how to port".
+- Writing validators: by design, rust-analyzer is very lax about the input. For
+  example, the lexer happily accepts unclosed strings. The idea is that there
+  should be a higher level visitor, which walks the syntax tree after parsing
+  and produces all the warnings. Alas, there's no such visitor yet :( Would you
+  like to write one? :)
 
-* Writing validators: by design, rust-analyzer is very lax about the
-  input. For example, the lexer happily accepts unclosed strings. The
-  idea is that there should be a higher level visitor, which walks the
-  syntax tree after parsing and produces all the warnings. Alas,
-  there's no such visitor yet :( Would you like to write one? :)
+- Creating tests: it would be tremendously helpful to read each of libsyntax and
+  rust-analyzer parser functions and crate a small separate test cases to cover
+  each and every edge case.
 
-* Creating tests: it would be tremendously helpful to read each of
-  libsyntax and rust-analyzer parser functions and crate a small separate
-  test cases to cover each and every edge case.
-
-* Building stuff with rust-analyzer: it would be really cool to compile
-  rust-analyzer to WASM and add *client side* syntax validation to rust
+- Building stuff with rust-analyzer: it would be really cool to compile
+  rust-analyzer to WASM and add _client side_ syntax validation to rust
   playground!
 
-
 Do take a look at the issue tracker.
 
-If you don't know where to start, or have *any* questions or suggestions,
-don't hesitate to chat at [Gitter](https://gitter.im/libsyntax2/Lobby)!
+If you don't know where to start, or have _any_ questions or suggestions, don't
+hesitate to chat at [Gitter]!
+
+# Code generation
+
+Some of the components of this repository are generated through automatic
+processes. These are outlined below:
+
+- `gen-kinds`: The kinds of tokens are reused in several places, so a generator
+  is used. This process uses [tera] to generate, using data in [grammar.ron],
+  the files:
+  - [ast/generated.rs][ast generated] in `ra_syntax` based on
+    [ast/generated.tera.rs][ast source]
+  - [syntax_kinds/generated.rs][syntax_kinds generated] in `ra_syntax` based on
+    [syntax_kinds/generated.tera.rs][syntax_kinds source]
+
+[libsyntax parser]:
+  https://github.com/rust-lang/rust/blob/6b99adeb11313197f409b4f7c4083c2ceca8a4fe/src/libsyntax/parse/parser.rs
+[kotlin parser]:
+  https://github.com/JetBrains/kotlin/blob/4d951de616b20feca92f3e9cc9679b2de9e65195/compiler/frontend/src/org/jetbrains/kotlin/parsing/KotlinParsing.java
+[gitter]: https://gitter.im/libsyntax2/Lobby
+[tera]: https://tera.netlify.com/
+[grammar.ron]: ./crates/ra_syntax/src/grammar.ron
+[ast generated]: ./crates/ra_syntax/src/ast/generated.rs
+[ast source]: ./crates/ra_syntax/src/ast/generated.tera.rs
+[syntax_kinds generated]: ./crates/ra_syntax/src/syntax_kinds/generated.rs
+[syntax_kinds source]: ./crates/ra_syntax/src/syntax_kinds/generated.tera.rs

crates/ra_syntax/src/ast/generated.rs

@@ -1,3 +1,6 @@
+// This file is automatically generated based on the file `./generated.rs.tera` when `cargo gen-kinds` is run
+// Do not edit manually
+
 use {
     ast,
     SyntaxNodeRef, AstNode,

crates/ra_syntax/src/ast/generated.rs.tera

@@ -1,3 +1,8 @@
+{# THIS File is not automatically generated:
+the below applies to the result of this template
+#}// This file is automatically generated based on the file `./generated.rs.tera` when `cargo gen-kinds` is run
+// Do not edit manually
+
 use {
     ast,
     SyntaxNodeRef, AstNode,

crates/ra_syntax/src/grammar.ron

@@ -1,3 +1,5 @@
+// Stores definitions which must be used in multiple places
+// See `cargo gen-kinds` (defined in crates/tools/src/main.rs)
 Grammar(
     single_byte_tokens: [
         [";", "SEMI"],
@@ -23,8 +25,9 @@ Grammar(
         ["^", "CARET"],
         ["%", "PERCENT"],
     ],
+    // TODO: Confirm surmision: the tokens which cannot be recorded in a single UTF-8 byte
     multi_byte_tokens: [
-        [".", "DOT"],
+        [".", "DOT"], // Note: DOT is here because <TODO: REASON>
         ["..", "DOTDOT"],
         ["...", "DOTDOTDOT"],
         ["..=", "DOTDOTEQ"],

crates/ra_syntax/src/syntax_kinds/generated.rs

@@ -1,3 +1,6 @@
+// This file is automatically generated based on the file `./generated.rs.tera` when `cargo gen-kinds` is run
+// Do not edit manually
+
 #![allow(bad_style, missing_docs, unreachable_pub)]
 #![cfg_attr(rustfmt, rustfmt_skip)]
 use super::SyntaxInfo;

crates/ra_syntax/src/syntax_kinds/generated.rs.tera

@@ -1,3 +1,8 @@
+{# THIS File is not automatically generated:
+the below applies to the result of this template
+#}// This file is automatically generated based on the file `./generated.rs.tera` when `cargo gen-kinds` is run
+// Do not edit manually
+
 #![allow(bad_style, missing_docs, unreachable_pub)]
 #![cfg_attr(rustfmt, rustfmt_skip)]
 use super::SyntaxInfo;