diff --git a/text/3502-cargo-script.md b/text/3502-cargo-script.md new file mode 100644 index 00000000000..55736f7f144 --- /dev/null +++ b/text/3502-cargo-script.md @@ -0,0 +1,1289 @@ +- Feature Name: `cargo-script` +- Start Date: 2023-03-31 +- Pre-RFC: [internals](https://internals.rust-lang.org/t/pre-rfc-cargo-script-for-everyone/18639) +- eRFC PR: [rust-lang/rfcs#3424](/~https://github.com/rust-lang/rfcs/pull/3424) +- RFC PR: [rust-lang/rfcs#3502](/~https://github.com/rust-lang/rfcs/pull/3502) +- Rust Issue: [rust-lang/cargo#12207](/~https://github.com/rust-lang/cargo/issues/12207) + +# Summary +[summary]: #summary + +This RFC adds support for single-file bin packages in cargo. +Single-file bin packages are rust source files with an embedded manifest and a +`main`. +These files will be accepted by cargo commands as `--manifest-path` just like `Cargo.toml` files. +`cargo` will be modified to accept `cargo .rs` as a shortcut to `cargo +run --manifest-path .rs`; +this allows placing `cargo` in a `#!` line for directly running these files. + +Support for single-file lib packages, publishing, and workspace support is +deferred out. + +Example: +````rust +#!/usr/bin/env cargo +--- +[dependencies] +clap = { version = "4.2", features = ["derive"] } +--- + +use clap::Parser; + +#[derive(Parser, Debug)] +#[clap(version)] +struct Args { + #[clap(short, long, help = "Path to config")] + config: Option, +} + +fn main() { + let args = Args::parse(); + println!("{:?}", args); +} +```` +```console +$ ./prog.rs --config file.toml +warning: `package.edition` is unspecified, defaulting to `2021` + Finished dev [unoptimized + debuginfo] target(s) in 0.06s + Running `/home/epage/.cargo/target/98/07dcd6510bdcec/debug/prog` +Args { config: Some("file.toml") } +``` + +See [`-Zscript`](https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#script) for a working implementation. + +# Motivation +[motivation]: #motivation + +**Collaboration:** + +When sharing reproduction cases, it is much easier when everything exists in a +single code snippet to copy/paste. +Alternatively, people will either leave off the manifest or underspecify the +details of it. + +This similarly makes it easier to share code samples with coworkers or in books +/ blogs when teaching. + +**Interoperability:** + +One angle to look at including something is if there is a single obvious +solution. +While there isn't in the case for single-file packages, there is enough of a +subset of one. +By standardizing that subset, we allow greater interoperability between +solutions +(e.g. [playground could gain support](https://users.rust-lang.org/t/call-for-contributors-to-the-rust-playground-for-upcoming-features/87110/14?u=epage)). +This would make it easier to collaborate.. + +**Prototyping:** + +Currently to prototype or try experiment with APIs or the language, you need to either +- Use the playground + - Can't access local resources + - Limited in the crates supported + - *Note:* there are alternatives to the playground that might have fewer + restrictions but are either less well known or have additional + complexities. +- Find a place to do `cargo new`, edit `Cargo.toml` and `main.rs` as necessary, and `cargo run` it, then delete it + - This is a lot of extra steps, increasing the friction to trying things out + - This will fail if you create in a place that `cargo` will think it should be a workspace member + +By having a single-file package, +- It is easier to setup and tear down these experiments, making it more likely to happen +- All crates will be available +- Local resources are available + +**One-Off Utilities:** + +It is fairly trivial to create a bunch of single-file bash or python scripts +into a directory and add it to the path. +Compare this to rust where +- `cargo new` each of the "scripts" into individual directories +- Create wrappers for each so you can access it in your path, passing `--manifest-path` to `cargo run` + +**Non-Goals:** + +With that said, this doesn't have to completely handle every use case for +Collaboration, Interoperability, Prototyping, or One-off Utilities. +Users can always scale up to normal packages with an explicit `Cargo.toml` file. + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +### Creating a New Package + +*(Adapted from [the cargo book](https://doc.rust-lang.org/cargo/guide/creating-a-new-project.html))* + +To start a new [package][def-package] with Cargo, create a file named `hello_world.rs`: +```rust +#!/usr/bin/env cargo + +fn main() { + println!("Hello, world!"); +} +``` + +Let's run it +```console +$ chmod +x hello_world.rs +$ ./hello_world.rs +warning: `package.edition` is unspecified, defaulting to `2021` + Finished dev [unoptimized + debuginfo] target(s) in 0.06s + Running `/home/epage/.cargo/target/98/07dcd6510bdcec/debug/hello_world` +Hello, world! +``` + +### Dependencies + +*(Adapted from [the cargo book](https://doc.rust-lang.org/cargo/guide/dependencies.html))* + +[crates.io] is the Rust community's central [*package registry*][def-package-registry] +that serves as a location to discover and download +[packages][def-package]. `cargo` is configured to use it by default to find +requested packages. + +#### Adding a dependency + +To depend on a library hosted on [crates.io], you modify `hello_world.rs`: +````rust +#!/usr/bin/env cargo +--- +[dependencies] +time = "0.1.12" +--- + +fn main() { + println!("Hello, world!"); +} +```` + +The data inside the `cargo` frontmatter is called a +[***manifest***][def-manifest], and it contains all of the metadata that Cargo +needs to compile your package. +This is written in the [TOML] format (pronounced /tɑməl/). + +`time = "0.1.12"` is the name of the [crate][def-crate] and a [SemVer] version +requirement. The [specifying +dependencies](https://doc.rust-lang.org/cargo/guide/../reference/specifying-dependencies.html) docs have more +information about the options you have here. + +If we also wanted to add a dependency on the `regex` crate, we would not need +to add `[dependencies]` for each crate listed. Here's what your whole +`hello_world.rs` file would look like with dependencies on the `time` and `regex` +crates: + +````rust +#!/usr/bin/env cargo +--- +[dependencies] +time = "0.1.12" +regex = "0.1.41" +--- + +fn main() { + let re = Regex::new(r"^\d{4}-\d{2}-\d{2}$").unwrap(); + println!("Did our date match? {}", re.is_match("2014-01-01")); +} +```` + +You can then re-run this and Cargo will fetch the new dependencies and all of their dependencies. You can see this by passing in `--verbose`: +```console +$ cargo --verbose ./hello_world.rs +warning: `package.edition` is unspecified, defaulting to `2021` + Updating crates.io index + Downloading memchr v0.1.5 + Downloading libc v0.1.10 + Downloading regex-syntax v0.2.1 + Downloading memchr v0.1.5 + Downloading aho-corasick v0.3.0 + Downloading regex v0.1.41 + Compiling memchr v0.1.5 + Compiling libc v0.1.10 + Compiling regex-syntax v0.2.1 + Compiling memchr v0.1.5 + Compiling aho-corasick v0.3.0 + Compiling regex v0.1.41 + Compiling hello_world v0.1.0 (file:///path/to/package/hello_world) + Finished dev [unoptimized + debuginfo] target(s) in 0.06s + Running `/home/epage/.cargo/target/98/07dcd6510bdcec/debug/hello_world` +Did our date match? true +``` + +## Package Layout + +*(Adapted from [the cargo book](https://doc.rust-lang.org/cargo/guide/project-layout.html))* + +When a single file is not enough, you can separately define a `Cargo.toml` file along with the `src/main.rs` file. Run +```console +$ cargo new hello_world --bin +``` + +We’re passing `--bin` because we’re making a binary program: if we +were making a library, we’d pass `--lib`. +This also initializes a new `git` repository by default. +If you don't want it to do that, pass `--vcs none`. + +Let’s check out what Cargo has generated for us: +```console +$ cd hello_world +$ tree . +. +├── Cargo.toml +└── src + └── main.rs + +1 directory, 2 files +``` +Unlike the `hello_world.rs`, a little more context is needed in `Cargo.toml`: +```toml +[package] +name = "hello_world" +version = "0.1.0" +edition = "2021" + +[dependencies] + +``` + +Cargo uses conventions for file placement to make it easy to dive into a new +Cargo [package][def-package]: + +```text +. +├── Cargo.lock +├── Cargo.toml +├── src/ +│   ├── lib.rs +│   ├── main.rs +│   └── bin/ +│ ├── named-executable.rs +│      ├── another-executable.rs +│      └── multi-file-executable/ +│      ├── main.rs +│      └── some_module.rs +├── benches/ +│   ├── large-input.rs +│   └── multi-file-bench/ +│   ├── main.rs +│   └── bench_module.rs +├── examples/ +│   ├── simple.rs +│   └── multi-file-example/ +│   ├── main.rs +│   └── ex_module.rs +└── tests/ + ├── some-integration-tests.rs + └── multi-file-test/ + ├── main.rs + └── test_module.rs +``` + +* `Cargo.toml` and `Cargo.lock` are stored in the root of your package (*package + root*). +* Source code goes in the `src` directory. +* The default library file is `src/lib.rs`. +* The default executable file is `src/main.rs`. + * Other executables can be placed in `src/bin/`. +* Benchmarks go in the `benches` directory. +* Examples go in the `examples` directory. +* Integration tests go in the `tests` directory. + +If a binary, example, bench, or integration test consists of multiple source +files, place a `main.rs` file along with the extra [*modules*][def-module] +within a subdirectory of the `src/bin`, `examples`, `benches`, or `tests` +directory. The name of the executable will be the directory name. + +You can learn more about Rust's module system in [the book][book-modules]. + +See [Configuring a target] for more details on manually configuring targets. +See [Target auto-discovery] for more information on controlling how Cargo +automatically infers target names. + +[book-modules]: https://doc.rust-lang.org/cargo/guide/../../book/ch07-00-managing-growing-projects-with-packages-crates-and-modules.html +[Configuring a target]: https://doc.rust-lang.org/cargo/guide/../reference/cargo-targets.html#configuring-a-target +[def-package]: https://doc.rust-lang.org/cargo/guide/../appendix/glossary.html#package '"package" (glossary entry)' +[Target auto-discovery]: https://doc.rust-lang.org/cargo/guide/../reference/cargo-targets.html#target-auto-discovery +[TOML]: https://toml.io/ +[crates.io]: https://crates.io/ +[SemVer]: https://semver.org +[def-crate]: https://doc.rust-lang.org/cargo/guide/../appendix/glossary.html#crate '"crate" (glossary entry)' +[def-package]: https://doc.rust-lang.org/cargo/guide/../appendix/glossary.html#package '"package" (glossary entry)' +[def-package-registry]: https://doc.rust-lang.org/cargo/guide/../appendix/glossary.html#package-registry '"package-registry" (glossary entry)' +[def-manifest]: https://doc.rust-lang.org/cargo/guide/../appendix/glossary.html#manifest '"manifest" (glossary entry)' + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +## Single-file packages + +In addition to today's multi-file packages (`Cargo.toml` file with other `.rs` +files), +we are adding the concept of single-file packages which may contain an +embedded manifest. +There is no required way to distinguish a single-file +`.rs` package from any other `.rs` file. + +A single-file package may contain an embedded manifest. +An embedded manifest is stored using `TOML` in rust "frontmatter", +a markdown code-fence with `cargo` at the start of the infostring at the top of +the file. + +Inferred / defaulted manifest fields: +- `package.name = ` +- `package.edition = ` to avoid always having to add an embedded + manifest at the cost of potentially breaking scripts on rust upgrades + - Warn when `edition` is unspecified. Since piped commands are run with `--quiet`, this may not show up. + - Based on feedback, we might add `cargo-` proxies to put in `#!` as a shorthand + - Based on feedback, we can switch to "edition is required as of edition" + +Note: As of [rust-lang/cargo#123](/~https://github.com/rust-lang/cargo/pull/12786), +when `package.version` is missing, +it gets defaulted to `0.0.0` and `package.publish` gets defaulted to `false`. + +Disallowed manifest fields: +- `[workspace]`, `[lib]`, `[[bin]]`, `[[example]]`, `[[test]]`, `[[bench]]` +- `package.workspace`, `package.build`, `package.links`, `package.publish`, `package.autobins`, `package.autoexamples`, `package.autotests`, `package.autobenches` + +Single-file packages maintain an out-of-line target directory by default. +This is implementation-defined. +Currently, it is `$CARGO_HOME/target/`. + +A single-file package is accepted by cargo commands as a `--manifest-path` +- Files are considered to have embedded manifest if they end with `.rs` or they lack an extension and are of type file +- This allows running `cargo test --manifest-path single.rs` +- `cargo add` and `cargo remove` may not support editing embedded manifests initially +- Path-dependencies may not refer to single-file packages at this time (they don't have a `lib` target anyways) + +Single-file packages will not be accepted as `path` or `git` dependencies. + +The lockfile for single-file packages will be placed in `CARGO_TARGET_DIR`. In +the future, when workspaces are supported, that will allow a user to have a +persistent lockfile. +We may also allow customizing the non-workspace lockfile location in the [future](#future-possibilities). + +## `cargo .rs` + +`cargo` is intended for putting in the `#!` for single-file packages: +```rust +#!/usr/bin/env cargo + +fn main() { + println!("Hello world"); +} +``` +- In contrast to `cargo run --manifest-path .rs`, `.cargo/config.toml` will not be loaded from the current-dir will instead be loaded from `CARGO_HOME`. + - This is inspired by `cargo install` though its logic is different: + - `cargo install --path ` will load config from `` + - All other `cargo install`s will load config from `CARGO_HOME` + - Unlike `cargo install`, we expect people to run single-file packages in unsafe locations, like temp directories or download directories, and don't want to pick up less trustworthy configs +- Like all cargo commands, including `cargo install`, the current-dir `rust-toolchain.toml` is respected ([cargo#7312](/~https://github.com/rust-lang/cargo/issues/7312)) +- `--release` is not passed in because the primary use case is for exploratory + programming, so the emphasis will be on build-time performance and debugging, + rather than runtime performance + +Most other flags and behavior will be similar to `cargo run`. + +The precedence for `cargo foo` will change from: +1. built-in commands +2. user aliases +3. third-party commands + +to: +1. built-in command xor manifest +2. user aliases +3. third-party commands + +To allow the xor, we enforce that +- we only assume the argument is a manifest if it has a `/` in it, ends with the `.rs` extension, or is `Cargo.toml` + - So `./build` can be used to run a script name `build` rather than the `cargo build` command +- no built-in command may look like an accepted manifest + +When the stdout or stderr of `cargo .rs` is not going to a terminal, cargo will assume `--quiet`. +Further work may be done to refine the output in interactive mode. + +# Drawbacks +[drawbacks]: #drawbacks + +The implicit content of the manifest will be unclear for users. +We can patch over this as best we can in documentation but the result won't be +ideal. +A user can workaround this with `cargo metadata --manifest-path .rs` +or `cargo read-manifest --manifest-path .rs` + +Like with all cargo packages, the `target/` directory grows unbounded. +This is made worse by them being out of the way and the scripts are likely to be short-lived, +removed without a `cargo clean --manifest-path foo`. +Some prior art include a cache GC but that is also to clean up the temp files +stored in other locations +(our temp files are inside the `target/` dir and should be rarer). +A GC for cargo is being tracked in [rust-lang/cargo#12633](/~https://github.com/rust-lang/cargo/issues/12633) + +With lockfile "hidden away" in the `target/`, +users might not be aware that they are using old dependencies. + +With `target/` including a hash of the script, moving the script throwsaway the build cache. +[cargo#5931](/~https://github.com/rust-lang/cargo/issues/5931) could help reduce this. + +Syntax is not reserved for `build.rs`, proc-maros, embedding +additional packages, or other functionality to be added later with the +assumption that if these features are needed, a user should be using a +multi-file package. +As stated in the Motivation, this doesn't have to perfectly cover every use +case that a `Cargo.toml` would. + +The precedence schema for `cargo foo` has limitations +- If your script has the same name as a built-in subcommand, then you have to prefix it with `./` +- If you browse a random repo and try to run one of your aliases or third-party commands, you could unintentionally get a local script instead. +- Similarly, new cargo commands could shadow user scripts +- If `PATH` is unset or set to an empty string, then running `build` will run `cargo build` and run the built-in `build` command rather than your script + - The likelihood of a script named the same as a cargo subcommand that is in the PATH or called in a strange way seems unlikely +- Calls to `execve` (and similar functions) don't rely on resolving via `PATH` so a call with `build` will run `cargo build` and run the built-in `build` command rather than your script + +This increases the maintenance and support burden for the cargo team, a team +that is already limited in its availability. + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +Initial guidelines for evaluating decisions: +- Single-file packages should have a first-class experience + - Provides a higher quality of experience (doesn't feel like a hack or tacked on) + - Transferable knowledge, whether experience, stackoverflow answers, etc + - Easier unassisted migration between single-file and multi-file packages + - The more the workflows deviate, the higher the maintenance and support costs for the cargo team + - Example implications: + - Workflows, like running tests, should be the same as multi-file packages rather than being bifurcated + - Manifest formats should be the same rather than using a specialized schema or data format +- Friction for starting a new single-file package should be minimal + - Easy to remember, minimal syntax so people are more likely to use it in + one-off cases, experimental or prototyping use cases without tool assistance + - Example implications: + - Embedded manifest is optional which also means we can't require users specifying `edition` + - See also the implications for first-class experience + - Workspaces for single-file packages should not be auto-discovered as that + will break unless the workspaces also owns the single-file package which + will break workflows for just creating a file anywhere to try out an + idea. +- Cargo/rustc diagnostics and messages (including `cargo metadata`) should be + in terms of single-file packages and not any temporary files + - Easier to understand the messages + - Provides a higher quality of experience (doesn't feel like a hack or tacked on) + - Example implications: + - Most likely, we'll need single-file packages to be understood directly by + rustc so cargo doesn't have to split out the `.rs` content into a temp + file that gets passed to cargo which will cause errors to point to the + wrong file + - Most likely, we'll want to muck with the errors returned by `toml_edit` + so we render manifest errors based on the original source code which will require accurate span information. + +## Misc + +- Rejected: Defaulting to `RUST_BACKTRACE=1` for `cargo foo.rs` runs + - Enabling backtraces provides more context to problems, much like Python scripts, which helps with experiments + - This comes at the cost of making things worse for scripts + - Decided against it to minimize special casing + - See also [t-cargo zulip thread](https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/Smarter.20.60RUST_BACKTRACE.60.20behavior.3F) +- The package name is slugified according the stricter `cargo new` validation + rules, making it consistent across platforms as some names are invalid on + some platforms + - See [rust-lang/cargo#12255](/~https://github.com/rust-lang/cargo/pull/12255) + +## Command-line / interactive evaluation + +The [`cargo-script`](https://crates.io/crates/cargo-script) family of tools has a single command for +- Run `.rs` files with embedded manifests +- Evaluate command-line arguments (`--expr`, `--loop`) + +This behavior (minus embedded manifests) mirrors what you might expect from a +scripting environment, minus a REPL. We could design this with the future possibility of a REPL. + +However +- The needs of `.rs` files and REPL / CLI args are different, e.g. where they get their dependency definitions +- A REPL is a lot larger of a problem, needing to pull in a lot of interactive behavior that is unrelated to `.rs` files +- A REPL for Rust is a lot more nebulous of a future possibility, making it pre-mature to design for it in mind + +Therefore, this RFC proposes we limit the scope of the new command to `cargo run` for single-file rust packages. + +## Naming +[naming]: #naming + +Considerations: +- The name should tie it back to `cargo` to convey that relationship +- The command that is run in a `#!` line should not require arguments (e.g. not + `#!/usr/bin/env cargo `) because it will fail. `env` treats the + rest of the line as the bin name, spaces included. You need to use `env -S` + but that isn't portable across all `env` implementations (e.g. busybox). +- Either don't have a name that looks like a cargo-plugin (e.g. not + `cargo-`) to avoid confusion or make it work (by default, `cargo + something` translates to `cargo-something something` which would be ambiguous + of whether `something` is a script or subcommand) + +Candidates +- `rust`: + - Would fit well for Rust scripting + - Would not make it clear that cargo is involved. +- `cargo-script`: + - Out of scope + - Verb preferred +- `cargo-shell`: + - Out of scope + - Verb preferred +- `cargo-run`: + - This would be shorthand for `cargo run --manifest-path