chore(book): adding help and contributing pages

README.md

@@ -69,108 +69,9 @@ To achieve those, **Logos**:
  }
 ```
 
-### Callbacks
-
-**Logos** can also call arbitrary functions whenever a pattern is matched,
-which can be used to put data into a variant:
-
-```rust
- use logos::{Logos, Lexer};
-
- // Note: callbacks can return `Option` or `Result`
- fn kilo(lex: &mut Lexer<Token>) -> Option<u64> {
-     let slice = lex.slice();
-     let n: u64 = slice[..slice.len() - 1].parse().ok()?; // skip 'k'
-     Some(n * 1_000)
- }
-
- fn mega(lex: &mut Lexer<Token>) -> Option<u64> {
-     let slice = lex.slice();
-     let n: u64 = slice[..slice.len() - 1].parse().ok()?; // skip 'm'
-     Some(n * 1_000_000)
- }
-
- #[derive(Logos, Debug, PartialEq)]
- #[logos(skip r"[ \t\n\f]+")]
- enum Token {
-     // Callbacks can use closure syntax, or refer
-     // to a function defined elsewhere.
-     //
-     // Each pattern can have its own callback.
-     #[regex("[0-9]+", |lex| lex.slice().parse().ok())]
-     #[regex("[0-9]+k", kilo)]
-     #[regex("[0-9]+m", mega)]
-     Number(u64),
- }
-
- fn main() {
-     let mut lex = Token::lexer("5 42k 75m");
-
-     assert_eq!(lex.next(), Some(Ok(Token::Number(5))));
-     assert_eq!(lex.slice(), "5");
-
-     assert_eq!(lex.next(), Some(Ok(Token::Number(42_000))));
-     assert_eq!(lex.slice(), "42k");
-
-     assert_eq!(lex.next(), Some(Ok(Token::Number(75_000_000))));
-     assert_eq!(lex.slice(), "75m");
-
-     assert_eq!(lex.next(), None);
- }
-```
-
-Logos can handle callbacks with following return types:
-
-| Return type            | Produces                                                                                            |
-|------------------------|-----------------------------------------------------------------------------------------------------|
-| `()`                   | `Ok(Token::Unit)`                                                                                   |
-| `bool`                 | `Ok(Token::Unit)` **or** `Err(<Token as Logos>::Error::default())`                                  |
-| `Result<(), E>`        | `Ok(Token::Unit)` **or** `Err(<Token as Logos>::Error::from(err))`                                  |
-| `T`                    | `Ok(Token::Value(T))`                                                                               |
-| `Option<T>`            | `Ok(Token::Value(T))` **or** `Err(<Token as Logos>::Error::default())`                              |
-| `Result<T, E>`         | `Ok(Token::Value(T))` **or** `Err(<Token as Logos>::Error::from(err))`                              |
-| [`Skip`]               | _skips matched input_                                                                               |
-| [`Filter<T>`]          | `Ok(Token::Value(T))` **or** _skips matched input_                                                  |
-| [`FilterResult<T, E>`] | `Ok(Token::Value(T))` **or** `Err(<Token as Logos>::Error::from(err))` **or** _skips matched input_ |
-
-Callbacks can be also used to do perform more specialized lexing in place
-where regular expressions are too limiting. For specifics look at
-`Lexer::remainder` and `Lexer::bump`.
-
-## Errors
-
-By default, **Logos** uses `()` as the error type, which means that it
-doesn't store any information about the error.
-This can be changed by using `#[logos(error = T)]` attribute on the enum.
-The type `T` can be any type that implements `Clone`, `PartialEq`,
-`Default`, `Debug` and `From<E>` for each callback's error type `E`.
-
-## Token disambiguation
-
-Rule of thumb is:
-
-+ Longer beats shorter.
-+ Specific beats generic.
-
-If any two definitions could match the same input, like `fast` and `[a-zA-Z]+`
-in the example above, it's the longer and more specific definition of `Token::Fast`
-that will be the result.
-
-This is done by comparing numeric priority attached to each definition. Every consecutive,
-non-repeating single byte adds 2 to the priority, while every range or regex class adds 1.
-Loops or optional blocks are ignored, while alternations count the shortest alternative:
-
-+ `[a-zA-Z]+` has a priority of 1 (lowest possible), because at minimum it can match a single byte to a class.
-+ `foobar` has a priority of 12.
-+ `(foo|hello)(bar)?` has a priority of 6, `foo` being its shortest possible match.
-
-If two definitions compute to the same priority and can match the same input **Logos** will
-fail to compile, point out the problematic definitions, and ask you to specify a manual
-priority for either of them.
-
-For example: `[abc]+` and `[cde]+` both can match sequences of `c`, and both have priority of 1.
-Turning the first definition to `#[regex("[abc]+", priority = 2)]` will allow for tokens
-to be disambiguated again, in this case all sequences of `c` will match `[abc]+`.
+For more examples and documentation, please refer to the
+[Logos handbook](https://maciejhirsz.github.io/logos/) or the
+[crate documentation](https://docs.rs/logos/latest/logos/).
 
 ## How fast?
 
@@ -191,6 +92,9 @@ test strings                           ... bench:         553 ns/iter (+/- 34) =
 **Logos** is very much a labor of love. If you find it useful, consider
 [getting me some coffee](/~https://github.com/sponsors/maciejhirsz). ☕
 
+If you'd like to contribute to Logos, then consider reading the
+[Contributing guide](https://maciejhirsz.github.io/logos/contributing).
+
 ## License
 
 This code is distributed under the terms of both the MIT license

book/src/SUMMARY.md

@@ -2,9 +2,7 @@
 
 + [Intro](./intro.md)
 + [Getting Started](./getting-started.md)
-+ [Examples](./examples.md)
-  + [Brainfuck interpreter](./examples/brainfuck.md)
-  + [JSON parser](./examples/json.md)
++ [Help](./getting-help.md)
 + [Attributes](./attributes.md)
   + [`#[logos]`](./attributes/logos.md)
   + [`#[error]`](./attributes/error.md)
@@ -13,3 +11,9 @@
 + [Using `Extras`](./extras.md)
 + [Using callbacks](./callbacks.md)
 + [Common regular expressions](./common-regex.md)
++ [Examples](./examples.md)
+  + [Brainfuck interpreter](./examples/brainfuck.md)
+  + [JSON parser](./examples/json.md)
++ [Contributing](./contributing.md)
+  + [Setup](./contributing/setup.md)
+  + [Internals](./contributing/internals.md)

book/src/contributing.md

@@ -0,0 +1,32 @@
+# Contributing
+
+If you are considering to contribute to Logos, then this place it for you!
+
+First, we really appreciate people that can help this project grow, and we
+would like to guide you through the standard contribution process.
+
+There are many ways to help us, and here is a short list of some of them:
+
++ fixing an [BUG](/~https://github.com/maciejhirsz/logos/labels/bug), by providing
+  a patch (or suggesting in the comments how one could fix it);
++ correcting some typos in the documentation, the book, or anywhere else;
++ raising an issue about a problem (i.e., 
+  [opening an issue](/~https://github.com/maciejhirsz/logos/issues/new) on GitHub);
++ proposing new features (either with 
+  [an issue](/~https://github.com/maciejhirsz/logos/issues/new) or 
+  [a pull request](/~https://github.com/maciejhirsz/logos/pulls) on GitHub);
++ or improving the documentation (either in the crate or in the book).
+
+In any case, GitHub is the place-to-go for anything related to contributing.
+
+Below, we provide a few help pages (or links) to contents that can help you
+understand Logos' internals and how you can create submit a contribution.
+
++ If you are new to GitHub or git, please consider reading those two guides:
+  + [GitHub’s Hello World](https://docs.github.com/en/get-started/quickstart/hello-world);
+  + and [GitHub Pull Request in 100 Seconds](https://www.youtube.com/watch?v=8lGpZkjnkt4&ab_channel=Fireship)
+    (video).
++ To setup and test your code locally, see the [Setup](./contributing/setup.md)
+  page.
++ To know a bit more how Logos works, check the
+  [Internals](./contributing/internals.md).

book/src/contributing/internals.md

@@ -0,0 +1,4 @@
+# Internals
+
+TODO: explaing the utility of each crate in the repository,
+and also how Logos creates lookup tables...

book/src/contributing/setup.md

@@ -0,0 +1,103 @@
+# Setup
+
+On this page, you will find all the information needed to run and test your
+own version of the Logos crate, locally.
+
+We assume you have basic knowledge with git and GitHub. If that is not the
+case, please refer to the link mentionned in [Contributing](./contributing.md).
+
+## Prerequisites
+
+You need to have both git and Rust installed on your computer,
+see intallation procedures:
+
++ for [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git);
++ and [Rust](https://www.rust-lang.org/tools/install).
+
+Once it's done, clone the Logos repository on your computer:
+
+```bash
+git clone /~https://github.com/maciejhirsz/logos.git
+```
+
+If you have a fork of this repository, make sure to clone it instead.
+
+Finally, launch a terminal (i.e., command-line) session and go to the
+`logos` directory.
+
+## Checking the code compiles
+
+A good way to see if you code can compile is to use the eponym command:
+
+```bash
+cargo check
+```
+
+## Formatting and linting your code
+
+Prior to suggesting changes in a pull request, it is important to both
+format your code:
+
+```bash
+cargo fmt
+```
+
+and check against Rust's linter:
+
+```bash
+cargo clippy
+```
+
+Make sure to run those frequently, otherwise your pull request will probably
+fail to pass the automated tests.
+
+## Testing your code
+
+A code that compiles isn't necessarily correct, and testing it against known
+cases is of good practice:
+
+```bash
+cargo test
+```
+
+## Building the documentation
+
+Logos' documentation needs to be built with Rust's nightly toolchain.
+
+You can install the latest nightly channel with:
+
+```bash
+rustup install nightly
+```
+
+Then, use the following command to build the documentation with a similar
+configuration to the one used by [docs.rs](https://docs.rs/logos/latest/logos/):
+
+```bash
+RUSTDOCFLAGS="--cfg docsrs" cargo +nightly doc \
+    --all-features \
+    -Zunstable-options \
+    -Zrustdoc-scrape-examples \
+    --no-deps \
+    --open \
+```
+
+
+## Building the book
+
+Logos' book can be built with mbBook.
+
+This tool can be installed with `cargo`:
+
+```bash
+cargo install mdbook
+```
+
+Then, you can build the book with:
+
+```bash
+mbook serve book --open
+```
+
+Any change in the `./book` folder will automatically trigger a new build,
+and the pages will be live-reloaded.

book/src/getting-help.md

@@ -0,0 +1,17 @@
+# Getting Help
+
+If you need help using Logos, there are three places you can go to depending
+on what you are looking for:
+
++ [this book](./) for a documented walk through Logo's usage, with detailed
+  examples, and more. A **must read** for any newcomer;
++ [the API documentation](https://docs.rs/logos/latest/logos/) to obtain precise
+  information about function signatures and what the Logos crate exposes in
+  terms of features;
++ and [GitHub issues](/~https://github.com/maciejhirsz/logos/issues) for anything
+  else that is not covered by any of the two above.
+
+Regarding [GitHub issues](/~https://github.com/maciejhirsz/logos/issues),
+it's highly recommended to first check if another issue, either open or closed,
+already covers the topic you are looking for. If not, then consider creating a
+new issue with necessary information about your question, problem or else.

logos/Cargo.toml

@@ -37,6 +37,7 @@ export_derive = ["logos-derive"]
 [[example]]
 name = "brainfuck"
 path = "examples/brainfuck.rs"
+doc-scrape-examples = true
 
 [[example]]
 name = "extras"
@@ -45,3 +46,4 @@ path = "examples/extras.rs"
 [[example]]
 name = "json"
 path = "examples/json.rs"
+doc-scrape-example = true

logos/src/lib.rs

@@ -17,7 +17,7 @@
 //! + [Unwinds loops](https://en.wikipedia.org/wiki/Loop_unrolling), and batches reads to minimize bounds checking.
 //! + Does all of that heavy lifting at compile time.
 //!
-//! See the Logos handbook <TODO URL> for additional documentation and usage examples.
+//! See the [Logos handbook](https://maciejhirsz.github.io/logos/) for additional documentation and usage examples.
 #![cfg_attr(not(feature = "std"), no_std)]
 #![cfg_attr(docsrs, feature(doc_auto_cfg))]
 #![warn(missing_docs)]