From 9fb9251204fdbacbe2e4739dbe66ffc45faa5752 Mon Sep 17 00:00:00 2001 From: Oliver Tale-Yazdi Date: Mon, 4 Sep 2023 14:45:33 +0200 Subject: [PATCH] Cleanup repo (a tiny bit) (#1382) * Delete stale adoc files Signed-off-by: Oliver Tale-Yazdi * Convert adoc to md Signed-off-by: Oliver Tale-Yazdi * Add adoc to gitignore Signed-off-by: Oliver Tale-Yazdi * Delete more random unmaintained files Signed-off-by: Oliver Tale-Yazdi * Markdown lint Signed-off-by: Oliver Tale-Yazdi --------- Signed-off-by: Oliver Tale-Yazdi --- .gitignore | 1 + cumulus/bridges/rustfmt.toml | 24 - polkadot/RELEASE.md | 52 -- polkadot/doc/release-checklist.md | 91 --- polkadot/node/service/README.adoc | 5 - polkadot/parachain/README.adoc | 5 - polkadot/primitives/README.adoc | 5 - polkadot/runtime/polkadot/README.adoc | 5 - polkadot/src/README.adoc | 5 - polkadot/statement-table/README.adoc | 5 - substrate/.maintain/getgoing.sh | 6 - substrate/.maintain/runtime-dep.py | 34 -- substrate/.maintain/update-deps.sh | 9 - ...ll-completion.adoc => shell-completion.md} | 23 +- substrate/client/cli/README.adoc | 6 - substrate/docs/README.adoc | 522 ------------------ substrate/docs/Structure.adoc | 121 ---- 17 files changed, 9 insertions(+), 910 deletions(-) delete mode 100644 cumulus/bridges/rustfmt.toml delete mode 100644 polkadot/RELEASE.md delete mode 100644 polkadot/doc/release-checklist.md delete mode 100644 polkadot/node/service/README.adoc delete mode 100644 polkadot/parachain/README.adoc delete mode 100644 polkadot/primitives/README.adoc delete mode 100644 polkadot/runtime/polkadot/README.adoc delete mode 100644 polkadot/src/README.adoc delete mode 100644 polkadot/statement-table/README.adoc delete mode 100644 substrate/.maintain/getgoing.sh delete mode 100755 substrate/.maintain/runtime-dep.py delete mode 100755 substrate/.maintain/update-deps.sh rename substrate/bin/node/cli/doc/{shell-completion.adoc => shell-completion.md} (75%) delete mode 100644 substrate/client/cli/README.adoc delete mode 100644 substrate/docs/README.adoc delete mode 100644 substrate/docs/Structure.adoc diff --git a/.gitignore b/.gitignore index bd7f34b481041..35e02e706b426 100644 --- a/.gitignore +++ b/.gitignore @@ -7,6 +7,7 @@ .local .vscode .wasm-binaries +*.adoc *.bin *.iml *.orig diff --git a/cumulus/bridges/rustfmt.toml b/cumulus/bridges/rustfmt.toml deleted file mode 100644 index 082150daf04ee..0000000000000 --- a/cumulus/bridges/rustfmt.toml +++ /dev/null @@ -1,24 +0,0 @@ -# Basic -hard_tabs = true -max_width = 100 -use_small_heuristics = "Max" -# Imports -imports_granularity = "Crate" -reorder_imports = true -# Consistency -newline_style = "Unix" -# Format comments -comment_width = 100 -wrap_comments = true -# Misc -chain_width = 80 -spaces_around_ranges = false -binop_separator = "Back" -reorder_impl_items = false -match_arm_leading_pipes = "Preserve" -match_arm_blocks = false -match_block_trailing_comma = true -trailing_comma = "Vertical" -trailing_semicolon = false -use_field_init_shorthand = true - diff --git a/polkadot/RELEASE.md b/polkadot/RELEASE.md deleted file mode 100644 index 196f27e595d6e..0000000000000 --- a/polkadot/RELEASE.md +++ /dev/null @@ -1,52 +0,0 @@ -# Polkadot Release Process - -## Branches -* release-candidate branch: The branch used for staging of the next release. Named like `release-v0.8.26` - -## Notes -* The release-candidate branch *must* be made in the `paritytech/polkadot` repo in order for release automation to work -correctly -* Any new pushes/merges to the release-candidate branch (for example, refs/heads/release-v0.8.26) will result in the rc -index being bumped (e.g., v0.8.26-rc1 to v0.8.26-rc2) and new wasm built. - -## Release workflow - -Below are the steps of the release workflow. Steps prefixed with NOACTION are automated and require no human action. - -1. To initiate the release process: - 1. branch master off to a release candidate branch: - `git checkout master; git pull; git checkout -b release-v0.8.26` - 1. In the [Substrate](/~https://github.com/paritytech/substrate) repo, check out the commit used by Polkadot (this can - be found using the following command in the *Polkadot* repo: `grep 'paritytech/substrate' Cargo.lock | grep -E - '[0-9a-f]{40}' | sort | uniq` - 1. Branch off this **Substrate** commit into its own branch: `git branch -b polkadot-v0.8.26; git push origin - refs/heads/polkadot-v0.8.26` - 1. In the **Polkadot** repository, use [diener](/~https://github.com/bkchr/diener/) to switch to this branch: `diener - update --branch "polkadot-v0.8.26" --substrate`. Update Cargo.lock (to do this, you can run `cargo build` and then - ctrl+c once it finishes fetching and begins compiling) - 1. Push the **Polkadot** `release-v0.8.26` branch to Github: `git push origin refs/heads/release-v0.8.26` -1. NOACTION: The current HEAD of the release-candidate branch is tagged `v0.8.26-rc1` -1. NOACTION: A draft release and runtime WASMs are created for this release-candidate automatically. A link to the draft - release will be linked in the internal Polkadot matrix channel. -1. NOACTION: A new Github issue is created containing a checklist of manual steps to be completed before we are - confident with the release. This will be linked in Matrix. -1. Complete the steps in the issue created in step 4, signing them off as completed -1. (optional) If a fix is required to the release-candidate: - 1. Merge the fix with `master` first - 1. Cherry-pick the commit from `master` to `release-v0.8.26`, fixing any merge conflicts. Try to avoid unnecessarily - bumping crates. - 1. Push the release-candidate branch to Github - this is now the new release- candidate - 1. Depending on the cherry-picked changes, it may be necessary to perform some or all of the manual tests again. - 1. If there are **Substrate** changes required, these should be cherry-picked to the Substrate `polkadot-v0.8.26` - branch and pushed, and the version of Substrate used in **Polkadot** updated using `cargo update -p sp-io` -1. Once happy with the release-candidate, tag the current top commit in the release candidate branch and push to Github: - `git tag -s -m 'v0.8.26' v0.8.26; git push --tags` -1. NOACTION: The HEAD of the `release` branch will be tagged with `v0.8.26`, and a final draft release will be created - on Github. - -## Security releases - -Occasionally there may be changes that need to be made to the most recently released version of Polkadot, without taking -*every* change to `master` since the last release. For example, in the event of a security vulnerability being found, -where releasing a fixed version is a matter of some expediency. In cases like this, the fix should first be merged with -master, cherry-picked to a branch forked from `release`, tested, and then finally merged with `release`. A sensible -versioning scheme for changes like this is `vX.Y.Z-1`. diff --git a/polkadot/doc/release-checklist.md b/polkadot/doc/release-checklist.md deleted file mode 100644 index 8c57791fed1d5..0000000000000 --- a/polkadot/doc/release-checklist.md +++ /dev/null @@ -1,91 +0,0 @@ - -# Notes - -## Burn In - -Ensure that Parity DevOps has run the new release on Westend, Kusama, and Polkadot validators for at least 12 hours -prior to publishing the release. - -## Build Artifacts - -Add any necessary assets to the release. They should include: - -* Linux binary -* GPG signature of the Linux binary -* SHA256 of binary -* Source code -* Wasm binaries of any runtimes - -## Release notes - -The release notes should list: - -* The priority of the release (i.e., how quickly users should upgrade) - this is based on the max priority of any - *client* changes. -* Which native runtimes and their versions are included -* The proposal hashes of the runtimes as built with [srtool](https://gitlab.com/chevdor/srtool) -* Any changes in this release that are still awaiting audit - -The release notes may also list: - -* Free text at the beginning of the notes mentioning anything important regarding this release -* Notable changes (those labelled with B[1-9]-* labels) separated into sections - -## Spec Version - -A runtime upgrade must bump the spec number. This may follow a pattern with the client release (e.g. runtime v12 -corresponds to v0.8.12, even if the current runtime is not v11). - -## Old Migrations Removed - -Any previous `on_runtime_upgrade` functions from old upgrades must be removed to prevent them from executing a second -time. The `on_runtime_upgrade` function can be found in `runtime//src/lib.rs`. - -## New Migrations - -Ensure that any migrations that are required due to storage or logic changes are included in the `on_runtime_upgrade` -function of the appropriate pallets. - -## Extrinsic Ordering - -Offline signing libraries depend on a consistent ordering of call indices and functions. Compare the metadata of the -current and new runtimes and ensure that the `module index, call index` tuples map to the same set of functions. In case -of a breaking change, increase `transaction_version`. - -To verify the order has not changed, you may manually start the following [Github -Action](/~https://github.com/paritytech/polkadot/actions/workflows/extrinsic-ordering-check-from-bin.yml). It takes around -a minute to run and will produce the report as artifact you need to manually check. - -The things to look for in the output are lines like: - * `[Identity] idx 28 -> 25 (calls 15)` - indicates the index for `Identity` has changed - * `[+] Society, Recovery` - indicates the new version includes 2 additional modules/pallets. - * If no indices have changed, every modules line should look something like `[Identity] idx 25 (calls 15)` - -Note: Adding new functions to the runtime does not constitute a breaking change as long as the indexes did not change. - -## Proxy Filtering - -The runtime contains proxy filters that map proxy types to allowable calls. If the new runtime contains any new calls, -verify that the proxy filters are up to date to include them. - -## Benchmarks - -There are three benchmarking machines reserved for updating the weights at release-time. To initialise a benchmark run -for each production runtime (`westend`, `kusama`, `polkadot`): -* Go to https://gitlab.parity.io/parity/polkadot/-/pipelines?page=1&scope=branches&ref=master -* Click the link to the last pipeline run for master -* Start each of the manual jobs: - * `update_westend_weights` - * `update_polkadot_weights` - * `update_kusama_weights` -* When these jobs have completed (it takes a few hours), a git PATCH file will be available to download as an artifact. -* On your local machine, branch off master -* Download the patch file and apply it to your branch with `git patch patchfile.patch` -* Commit the changes to your branch and submit a PR against master -* The weights should be (Currently manually) checked to make sure there are no big outliers (i.e., twice or half the - weight). - -## Polkadot JS - -Ensure that a release of [Polkadot JS API](/~https://github.com/polkadot-js/api) contains any new types or interfaces -necessary to interact with the new runtime. diff --git a/polkadot/node/service/README.adoc b/polkadot/node/service/README.adoc deleted file mode 100644 index 2196d5467806c..0000000000000 --- a/polkadot/node/service/README.adoc +++ /dev/null @@ -1,5 +0,0 @@ - -= Polkadot Service - -placeholder -//TODO Write content :) (/~https://github.com/paritytech/polkadot/issues/159) diff --git a/polkadot/parachain/README.adoc b/polkadot/parachain/README.adoc deleted file mode 100644 index 8650919e64ec4..0000000000000 --- a/polkadot/parachain/README.adoc +++ /dev/null @@ -1,5 +0,0 @@ - -= Polkadot Parachain - -placeholder -//TODO Write content :) (/~https://github.com/paritytech/polkadot/issues/159) diff --git a/polkadot/primitives/README.adoc b/polkadot/primitives/README.adoc deleted file mode 100644 index 0e5c9412f0029..0000000000000 --- a/polkadot/primitives/README.adoc +++ /dev/null @@ -1,5 +0,0 @@ - -= Polkadot primitives - -placeholder -//TODO Write content :) (/~https://github.com/paritytech/polkadot/issues/159) diff --git a/polkadot/runtime/polkadot/README.adoc b/polkadot/runtime/polkadot/README.adoc deleted file mode 100644 index 33373310819f9..0000000000000 --- a/polkadot/runtime/polkadot/README.adoc +++ /dev/null @@ -1,5 +0,0 @@ - -= Polkadot Runtime - -placeholder -//TODO Write content :) (/~https://github.com/paritytech/polkadot/issues/159) diff --git a/polkadot/src/README.adoc b/polkadot/src/README.adoc deleted file mode 100644 index 4ec8e18d8afe9..0000000000000 --- a/polkadot/src/README.adoc +++ /dev/null @@ -1,5 +0,0 @@ - -= Polkadot Src - -placeholder -//TODO Write content :) (/~https://github.com/paritytech/polkadot/issues/159) diff --git a/polkadot/statement-table/README.adoc b/polkadot/statement-table/README.adoc deleted file mode 100644 index a4da4dee80ff5..0000000000000 --- a/polkadot/statement-table/README.adoc +++ /dev/null @@ -1,5 +0,0 @@ - -= Polkadot Statement table - -placeholder -//TODO Write content :) (/~https://github.com/paritytech/polkadot/issues/159) diff --git a/substrate/.maintain/getgoing.sh b/substrate/.maintain/getgoing.sh deleted file mode 100644 index 98f360837d04a..0000000000000 --- a/substrate/.maintain/getgoing.sh +++ /dev/null @@ -1,6 +0,0 @@ -/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" -brew install openssl cmake -curl https://sh.rustup.rs -sSf | sh -source ~/.cargo/env -cargo install --git /~https://github.com/paritytech/substrate subkey -cargo install --git /~https://github.com/paritytech/substrate substrate diff --git a/substrate/.maintain/runtime-dep.py b/substrate/.maintain/runtime-dep.py deleted file mode 100755 index 3198bb3e2669e..0000000000000 --- a/substrate/.maintain/runtime-dep.py +++ /dev/null @@ -1,34 +0,0 @@ -#!/usr/bin/env python3 - -# To run this script, you need to install the 'toml' python package and install the 'graphviz' package: -# pip install toml -# sudo apt-get install graphviz -# the first parameter is the runtime folder -# python ./.maintain/runtime-dep.py ./substrate/runtime | dot -Tpng -o output.png -import sys -import os -import toml - -if len(sys.argv) != 2: - print("needs the runtime folder.") - sys.exit(-1) - -runtime_dir = sys.argv[1] - -files = [os.path.join(runtime_dir, f, 'Cargo.toml') for f in os.listdir(runtime_dir) if os.path.isfile(os.path.join(runtime_dir, f, 'Cargo.toml')) and f != 'example'] - -print("digraph G {") - - -PREFIX = "substrate-runtime-" - -for f in files: - manifest = toml.load(f) - - package_name = manifest['package']['name'] - deps = [d for d in manifest['dependencies'].keys() if d.startswith(PREFIX)] - - for d in deps: - print(" \"{}\" -> \"{}\";".format(package_name, d)) - -print("}") diff --git a/substrate/.maintain/update-deps.sh b/substrate/.maintain/update-deps.sh deleted file mode 100755 index cd6b7c853825e..0000000000000 --- a/substrate/.maintain/update-deps.sh +++ /dev/null @@ -1,9 +0,0 @@ -#!/bin/sh -- -set -eu -case $0 in - (/*) dir=${0%/*}/;; - (*/*) dir=./${0%/*};; - (*) dir=.;; -esac - -find "$dir/.." -name Cargo.lock -execdir cargo update \; diff --git a/substrate/bin/node/cli/doc/shell-completion.adoc b/substrate/bin/node/cli/doc/shell-completion.md similarity index 75% rename from substrate/bin/node/cli/doc/shell-completion.adoc rename to substrate/bin/node/cli/doc/shell-completion.md index 168f00994fb2d..3f009a04373d9 100644 --- a/substrate/bin/node/cli/doc/shell-completion.adoc +++ b/substrate/bin/node/cli/doc/shell-completion.md @@ -1,12 +1,11 @@ +# Shell completion -== Shell completion - -The Substrate cli command supports shell auto-completion. For this to work, you will need to run the completion script matching your build and system. +The Substrate cli command supports shell auto-completion. For this to work, you will need to run the +completion script matching your build and system. Assuming you built a release version using `cargo build --release` and use `bash` run the following: -[source, shell] -source target/release/completion-scripts/substrate.bash +`source target/release/completion-scripts/substrate.bash` You can find completion scripts for: - bash @@ -17,25 +16,19 @@ You can find completion scripts for: To make this change persistent, you can proceed as follows: -.First install - -[source, shell] ----- +```shell COMPL_DIR=$HOME/.completion mkdir -p $COMPL_DIR cp -f target/release/completion-scripts/substrate.bash $COMPL_DIR/ echo "source $COMPL_DIR/substrate.bash" >> $HOME/.bash_profile source $HOME/.bash_profile ----- - -.Update +``` When you build a new version of Substrate, the following will ensure your auto-completion script matches the current binary: -[source, shell] ----- +```shell COMPL_DIR=$HOME/.completion mkdir -p $COMPL_DIR cp -f target/release/completion-scripts/substrate.bash $COMPL_DIR/ source $HOME/.bash_profile ----- +``` diff --git a/substrate/client/cli/README.adoc b/substrate/client/cli/README.adoc deleted file mode 100644 index fc58908fdf23d..0000000000000 --- a/substrate/client/cli/README.adoc +++ /dev/null @@ -1,6 +0,0 @@ - -= Substrate CLI - -Substrate CLI library - -include::doc/shell-completion.adoc[] diff --git a/substrate/docs/README.adoc b/substrate/docs/README.adoc deleted file mode 100644 index 3537e346a66e1..0000000000000 --- a/substrate/docs/README.adoc +++ /dev/null @@ -1,522 +0,0 @@ -= Substrate -:Author: Substrate developers -:Revision: 0.2.0 -:toc: -:sectnums: - -== Intro in one sentence - -Substrate is a next-generation framework for blockchain innovation. - -== Description - -At its heart, Substrate is a combination of three technologies: https://webassembly.org/[WebAssembly], https://libp2p.io/[Libp2p] and GRANDPA Consensus. About GRANDPA, see this https://hackmd.io/Jd0byWX0RiqFiXUVC78Bdw?view#GRANDPA[definition], https://medium.com/polkadot-network/grandpa-block-finality-in-polkadot-an-introduction-part-1-d08a24a021b5[introduction] and /~https://github.com/w3f/consensus/blob/master/pdf/grandpa.pdf[formal specification]. It is both a library for building new blockchains and a "skeleton key" of a blockchain client, able to synchronize to any Substrate-based chain. - -Substrate chains have three distinct features that make them "next-generation": a dynamic, self-defining state-transition function; light-client functionality from day one; and a progressive consensus algorithm with fast block production and adaptive, definite finality. The STF, encoded in WebAssembly, is known as the "runtime". This defines the `execute_block` function, and can specify everything from the staking algorithm, transaction semantics, logging mechanisms and procedures for replacing any aspect of itself or of the blockchain's state ("governance"). Because the runtime is entirely dynamic all of these can be switched out or upgraded at any time. A Substrate chain is very much a "living organism". - -See also https://www.parity.io/what-is-substrate/. - -== Usage - -Substrate is still an early stage project, and while it has already been used as the basis of major projects like Polkadot, using it is still a significant undertaking. In particular, you should have a good knowledge of blockchain concepts and basic cryptography. Terminology like header, block, client, hash, transaction and signature should be familiar. At present you will need a working knowledge of Rust to be able to do anything interesting (though eventually, we aim for this not to be the case). - -Substrate is designed for use in one of three ways: - -**1. Trivial**: By running the Substrate binary `substrate` and configuring it with a genesis block that includes the current demonstration runtime. In this case, you just build Substrate, configure a JSON file, and launch your own blockchain. This affords you the least amount of customizability, primarily allowing you to change the genesis parameters of the various included runtime modules such as balances, staking, block-period, fees, and governance. - -**2. Modular**: By hacking together pallets built with Substrate FRAME into a new runtime and possibly altering or reconfiguring the Substrate client's block authoring logic. This affords you a very large amount of freedom over your blockchain's logic, letting you change data types, add or remove modules, and crucially, add your own modules. Much can be changed without touching the block authoring logic (since it is generic). If this is the case, then the existing Substrate binary can be used for block authoring and syncing. If the block authoring logic needs to be tweaked, then a new, altered block authoring binary must be built as a separate project and used by validators. This is how the Polkadot relay chain is built and should suffice for almost all circumstances in the near to mid-term. - -**3. Generic**: The entire FRAME can be ignored and the entire runtime designed and implemented from scratch. If desired, this can be done in a language other than Rust, provided it can target WebAssembly. If the runtime can be made compatible with the existing client's block authoring logic, then you can simply construct a new genesis block from your Wasm blob and launch your chain with the existing Rust-based Substrate client. If not, then you'll need to alter the client's block authoring logic accordingly. This is probably a useless option for most projects right now, but provides complete flexibility allowing for a long-term, far-reaching upgrade path for the Substrate paradigm. - -=== The Basics of Substrate - -Substrate is a blockchain platform with a completely generic state transition function. That said, it does come with both standards and conventions (particularly regarding the Runtime Module Library) regarding underlying data structures. Roughly speaking, these core data types correspond to +trait+s in terms of the actual non-negotiable standard and generic +struct+s in terms of the convention. - -``` -Header := Parent + ExtrinsicsRoot + StorageRoot + Digest -Block := Header + Extrinsics + Justifications -``` - -=== Extrinsics - -Extrinsics in Substrate are pieces of information from "the outside world" that are contained in the blocks of the chain. You might think "ahh, that means *transactions*": in fact, no. Extrinsics fall into two broad categories of which only one is *transactions*. The other is known as *inherents*. The difference between these two is that transactions are signed and gossiped on the network and can be deemed useful *per se*. This fits the mold of what you would call transactions in Bitcoin or Ethereum. - -Inherents, meanwhile, are not passed on the network and are not signed. They represent data which describes the environment but which cannot call upon anything to prove it such as a signature. Rather they are assumed to be "true" simply because a sufficiently large number of validators have agreed on them being reasonable. - -To give an example, there is the timestamp inherent, which sets the current timestamp of the block. This is not a fixed part of Substrate, but does come as part of FRAME to be used as desired. No signature could fundamentally prove that a block were authored at a given time in quite the same way that a signature can "prove" the desire to spend some particular funds. Rather, it is the business of each validator to ensure that they believe the timestamp is set to something reasonable before they agree that the block candidate is valid. - -Other examples include the parachain-heads extrinsic in Polkadot and the "note-missed-proposal" extrinsic used in FRAME to determine and punish or deactivate offline validators. - - -=== Runtime and API - -Substrate chains all have a runtime. The runtime is a WebAssembly "blob" that includes a number of entry-points. Some entry-points are required as part of the underlying Substrate specification. Others are merely convention and required for the default implementation of the Substrate client to be able to author blocks. - -If you want to develop a chain with Substrate, you will need to implement the `Core` trait. This `Core` trait generates an API with the minimum necessary functionality to interact with your runtime. A special macro is provided called `impl_runtime_apis!` that help you implement runtime API traits. All runtime API trait implementations need to be done in one call of the `impl_runtime_apis!` macro. All parameters and return values need to implement https://crates.io/crates/parity-codec[`parity-codec`] to be encodable and decodable. - -Here's a snippet of the Polkadot API implementation as of PoC-3: - -```rust -impl_runtime_apis! { - impl client_api::Core for Runtime { - fn version() -> RuntimeVersion { - VERSION - } - - fn execute_block(block: Block) { - Executive::execute_block(block) - } - - fn initialize_block(header: ::Header) { - Executive::initialize_block(&header) - } - } - // ---snip--- -} -``` - - -=== Inherent Extrinsics - -Substrate FRAME includes functionality for timestamps and slashing. If used, these rely on "trusted" external information being passed in via inherent extrinsics. The Substrate reference block authoring client software will expect to be able to call into the runtime API with collated data (in the case of the reference Substrate authoring client, this is merely the current timestamp and which nodes were offline) in order to return the appropriate extrinsics ready for inclusion. If new inherent extrinsic types and data are to be used in a modified runtime, then it is this function (and its argument type) that would change. - -=== Block-authoring Logic - -In Substrate, there is a major distinction between blockchain *syncing* and block *authoring* ("authoring" is a more general term for what is called "mining" in Bitcoin). The first case might be referred to as a "full node" (or "light node" - Substrate supports both): authoring necessarily requires a synced node and, therefore, all authoring clients must necessarily be able to synchronize. However, the reverse is not true. The primary functionality that authoring nodes have which is not in "sync nodes" is threefold: transaction queue logic, inherent transaction knowledge and BFT consensus logic. BFT consensus logic is provided as a core element of Substrate and can be ignored since it is only exposed in the SDK under the `authorities()` API entry. - -Transaction queue logic in Substrate is designed to be as generic as possible, allowing a runtime to express which transactions are fit for inclusion in a block through the `initialize_block` and `apply_extrinsic` calls. However, more subtle aspects like prioritization and replacement policy must currently be expressed "hard coded" as part of the blockchain's authoring code. That said, Substrate's reference implementation for a transaction queue should be sufficient for an initial chain implementation. - -Inherent extrinsic knowledge is again somewhat generic, and the actual construction of the extrinsics is, by convention, delegated to the "soft code" in the runtime. If ever there needs to be additional extrinsic information in the chain, then both the block authoring logic will need to be altered to provide it into the runtime and the runtime's `inherent_extrinsics` call will need to use this extra information in order to construct any additional extrinsic transactions for inclusion in the block. - -== Roadmap - -=== So far - -- 0.1 "PoC-1": PBFT consensus, Wasm runtime engine, basic runtime modules. -- 0.2 "PoC-2": Libp2p - -=== In progress - -- AfG consensus -- Improved PoS -- Smart contract runtime module - -=== The future - -- Splitting out runtime modules into separate repo -- Introduce substrate executable (the skeleton-key runtime) -- Introduce basic but extensible transaction queue and block-builder and place them in the executable. -- DAO runtime module -- Audit - -== Trying out Substrate Node - -Substrate Node is Substrate's pre-baked blockchain client. You can run a development node locally or configure a new chain and launch your own global testnet. - -=== On Mac and Ubuntu - -To get going as fast as possible, there is a simple script that installs all required dependencies and installs Substrate into your path. Just open a terminal and run: - -[source, shell] ----- -curl https://getsubstrate.io -sSf | bash ----- - -You can start a local Substrate development chain with running `substrate --dev`. - -To create your own global network/cryptocurrency, you'll need to make a new Substrate Node chain specification file ("chainspec"). - -First let's get a template chainspec that you can edit. We'll use the "staging" chain, a sort of default chain that the node comes pre-configured with: - -[source, shell] ----- -substrate build-spec --chain=staging > ~/chainspec.json ----- - -Now, edit `~/chainspec.json` in your editor. There are a lot of individual fields for each module, and one very large one which contains the WebAssembly code blob for this chain. The easiest field to edit is the block `period`. Change it to 10 (seconds): - -[source, json] ----- - "timestamp": { - "minimumPeriod": 10 - }, ----- - -Now with this new chainspec file, you can build a "raw" chain definition for your new chain: - -[source, shell] ----- -substrate build-spec --chain ~/chainspec.json --raw > ~/mychain.json ----- - -This can be fed into Substrate: - -[source, shell] ----- -substrate --chain ~/mychain.json ----- - -It won't do much until you start producing blocks though, so to do that you'll need to use the `--validator` option together with passing the seed for the account(s) that is configured to be the initial authorities: - -[source, shell] ----- -substrate --chain ~/mychain.json --validator ----- - -You can distribute `mychain.json` so that everyone can synchronize and (depending on your authorities list) validate on your chain. - - -== Building - -=== Hacking on Substrate - -If you'd actually like to hack on Substrate, you can just grab the source code and -build it. Ensure you have Rust and the support software installed: - -==== Linux and Mac - -For Unix-based operating systems, you should run the following commands: - -[source, shell] ----- -curl https://sh.rustup.rs -sSf | sh - -rustup update nightly -rustup target add wasm32-unknown-unknown --toolchain nightly -rustup update stable ----- - -You will also need to install the following packages: - - - Linux: -[source, shell] -sudo apt install cmake pkg-config libssl-dev git clang libclang-dev llvm - -- Linux on ARM: -`rust-lld` is required for linking wasm, but is missing on non Tier 1 platforms. -So, use this /~https://github.com/Plume-org/Plume/blob/master/script/wasm-deps.sh[script] -to build `lld` and create the symlink `/usr/bin/rust-lld` to the build binary. - - - Mac: -[source, shell] -brew install cmake pkg-config openssl git llvm - -To finish installation of Substrate, jump down to <>. - -==== Windows - -If you are trying to set up Substrate on Windows, you should do the following: - -1. First, you will need to download and install "Build Tools for Visual Studio:" - - * You can get it at this link: https://aka.ms/buildtools - * Run the installation file: `vs_buildtools.exe` - * Please ensure the Windows 10 SDK component is included when installing the Visual C++ Build Tools. - * image:https://i.imgur.com/zayVLmu.png[image] - * Restart your computer. - -2. Next, you need to install Rust: - - * Detailed instructions are provided by the https://doc.rust-lang.org/book/ch01-01-installation.html#installing-rustup-on-windows[Rust Book]. - * Download from: https://www.rust-lang.org/tools/install - * Run the installation file: `rustup-init.exe` - > Note that it should not prompt you to install vs_buildtools since you did it in step 1. - * Choose "Default Installation." - * To get started, you need Cargo's bin directory (%USERPROFILE%\.cargo\bin) in your PATH environment variable. Future applications will automatically have the correct environment, but you may need to restart your current shell. - -3. Then, you will need to run some commands in CMD to set up your Wasm Build Environment: - - rustup update nightly - rustup update stable - rustup target add wasm32-unknown-unknown --toolchain nightly - -4. Then, you need to install LLVM: https://releases.llvm.org/download.html - -5. Next, you need to install OpenSSL, which we will do with `vcpkg`: - - mkdir \Tools - cd \Tools - git clone /~https://github.com/Microsoft/vcpkg.git - cd vcpkg - .\bootstrap-vcpkg.bat - .\vcpkg.exe install openssl:x64-windows-static - -6. After, you need to add OpenSSL to your System Variables. Note that in order for the following commands to work, you need to use Windows Powershell: - - $env:OPENSSL_DIR = 'C:\Tools\vcpkg\installed\x64-windows-static' - $env:OPENSSL_STATIC = 'Yes' - [System.Environment]::SetEnvironmentVariable('OPENSSL_DIR', $env:OPENSSL_DIR, [System.EnvironmentVariableTarget]::User) - [System.Environment]::SetEnvironmentVariable('OPENSSL_STATIC', $env:OPENSSL_STATIC, [System.EnvironmentVariableTarget]::User) - -7. Finally, you need to install `cmake`: https://cmake.org/download/ - -==== Docker - -You can use /~https://github.com/paritytech/scripts/tree/master/dockerfiles/ci-linux[Parity CI docker image] with all necessary dependencies to build Substrate: - -[source, shell] ----- -#run it in the folder with the Substrate source code -docker run --rm -it -w /shellhere/substrate \ - -v $(pwd):/shellhere/substrate \ - paritytech/ci-linux:production ----- - -You can find necessary cargo commands in <> - -==== Shared Steps - -Then, grab the Substrate source code: - -[source, shell] ----- -git clone /~https://github.com/paritytech/substrate.git -cd substrate ----- - -Then build the code: - -[source, shell] ----- -cargo build # Builds all native code ----- - -You can run all the tests if you like: - -[source, shell] -cargo test --all - -Or just run the tests of a specific package (i.e. `cargo test -p pallet-assets`) - -You can start a development chain with: - -[source, shell] -cargo run --release -- --dev - -Detailed logs may be shown by running the node with the following environment variables set: `RUST_LOG=debug RUST_BACKTRACE=1 cargo run --release \-- --dev`. - -If you want to see the multi-node consensus algorithm in action locally, then you can create a local testnet with two validator nodes for Alice and Bob, who are the initial authorities of the genesis chain specification that have been endowed with a testnet DOTs. We'll give each node a name and expose them so they are listed on link:https://telemetry.polkadot.io/#/Local%20Testnet[Telemetry]. You'll need two terminal windows open. - -We'll start Alice's Substrate node first on default TCP port 30333 with their chain database stored locally at `/tmp/alice`. The Bootnode ID of Alice's node is `QmRpheLN4JWdAnY7HGJfWFNbfkQCb6tFf4vvA6hgjMZKrR`, which is generated from the `--node-key` value that we specify below: - -[source, shell] -cargo run --release \-- \ - --base-path /tmp/alice \ - --chain=local \ - --alice \ - --node-key 0000000000000000000000000000000000000000000000000000000000000001 \ - --telemetry-url 'ws://telemetry.polkadot.io:1024 0' \ - --validator - -In the second terminal, we'll run the following to start Bob's Substrate node on a different TCP port of 30334, and with their chain database stored locally at `/tmp/bob`. We'll specify a value for the `--bootnodes` option that will connect Bob's node to Alice's Bootnode ID on TCP port 30333: - -[source, shell] -cargo run --release \-- \ - --base-path /tmp/bob \ - --bootnodes /ip4/127.0.0.1/tcp/30333/p2p/QmRpheLN4JWdAnY7HGJfWFNbfkQCb6tFf4vvA6hgjMZKrR \ - --chain=local \ - --bob \ - --port 30334 \ - --telemetry-url 'ws://telemetry.polkadot.io:1024 0' \ - --validator - -Additional Substrate CLI usage options are available and may be shown by running `cargo run \-- --help`. - -[[flaming-fir]] -=== Joining the Flaming Fir Testnet - -Flaming Fir is the new testnet for Substrate master (2.0) to test the latest development features. Please note that master is not compatible with the BBQ Birch, Charred Cherry, Dried Danta or Emberic Elm testnets. Ensure you have the dependencies listed above before compiling. - -Since Flaming Fir is targeting the master branch we make absolutely no guarantees of stability and/or persistence of the network. We might reset the chain at any time if it is necessary to deploy new changes. Currently, the validators are running with a client built from `d013bd900`, if you build from this commit you should be able to successfully sync, later commits may not work as new breaking changes may be introduced in master. - -Latest known working version: `a2a0eb5398d6223e531455b4c155ef053a4a3a2b` - -[source, shell] ----- -git clone /~https://github.com/paritytech/substrate.git -cd substrate -git checkout -b flaming-fir a2a0eb5398d6223e531455b4c155ef053a4a3a2b ----- - -You can run the tests if you like: - -[source, shell] -cargo test --all - -Start your node: - -[source, shell] -cargo run --release \-- - -To see a list of command line options, enter: - -[source, shell] -cargo run --release \-- --help - -For example, you can choose a custom node name: - -[source, shell] -cargo run --release \-- --name my_custom_name - -If you are successful, you will see your node syncing at https://telemetry.polkadot.io/#/Flaming%20Fir - -=== Joining the Emberic Elm Testnet - -Emberic Elm is the testnet for Substrate 1.0. Please note that 1.0 is not compatible with the BBQ Birch, Charred Cherry, Dried Danta or Flaming Fir testnets. -In order to join the Emberic Elm testnet you should build from the `v1.0` branch. Ensure you have the dependencies listed above before compiling. - -[source, shell] ----- -git clone /~https://github.com/paritytech/substrate.git -cd substrate -git checkout -b v1.0 origin/v1.0 ----- - -You can then follow the same steps for building and running as described above in <>. - -== Key management - -Keys in Substrate are stored in the keystore in the file system. To store keys into this keystore, -you need to use one of the two provided RPC calls. If your keys are encrypted or should be encrypted -by the keystore, you need to provide the key using one of the cli arguments `--password`, -`--password-interactive` or `--password-filename`. - -=== Recommended RPC call - -For most users who want to run a validator node, the `author_rotateKeys` RPC call is sufficient. -The RPC call will generate `N` Session keys for you and return their public keys. `N` is the number -of session keys configured in the runtime. The output of the RPC call can be used as input for the -`session::set_keys` transaction. - -``` -curl -H 'Content-Type: application/json' --data '{ "jsonrpc":"2.0", "method":"author_rotateKeys", "id":1 }' localhost:9933 -``` - -=== Advanced RPC call - -If the Session keys need to match a fixed seed, they can be set individually key by key. The RPC call -expects the key seed and the key type. The key types supported by default in Substrate are listed -/~https://github.com/paritytech/substrate/blob/master/core/primitives/src/crypto.rs#L767[here], but the -user can declare any key type. - -``` -curl -H 'Content-Type: application/json' --data '{ "jsonrpc":"2.0", "method":"author_insertKey", "params":["KEY_TYPE", "SEED", "PUBLIC"],"id":1 }' localhost:9933 -``` - -`KEY_TYPE` - needs to be replaced with the 4-character key type identifier. -`SEED` - is the seed of the key. -`PUBLIC` - public key for the given key. - -== Documentation - -=== Viewing documentation for Substrate packages - -You can generate documentation for a Substrate Rust package and have it automatically open in your web browser using https://doc.rust-lang.org/rustdoc/what-is-rustdoc.html#using-rustdoc-with-cargo[rustdoc with Cargo], -(of the The Rustdoc Book), by running the following command: - -``` -cargo doc --package --open -``` - -Replacing `` with one of the following (i.e. `cargo doc --package substrate --open`): - -* All Substrate Packages -[source, shell] -substrate -* Substrate Core -[source, shell] -substrate, substrate-cli, substrate-client, substrate-client-db, -substrate-consensus-common, substrate-consensus-rhd, -substrate-executor, substrate-finality-grandpa, substrate-keyring, substrate-keystore, substrate-network, -substrate-network-libp2p, substrate-primitives, substrate-rpc, substrate-rpc-servers, -substrate-serializer, substrate-service, substrate-service-test, substrate-state-db, -substrate-state-machine, substrate-telemetry, substrate-test-client, -substrate-test-runtime, substrate-transaction-graph, sp-transaction-pool, -substrate-trie -* Substrate Runtime -[source, shell] -sr-api, sr-io, sr-primitives, sr-sandbox, sr-std, sr-version -* FRAME Core -[source, shell] -frame-metadata, frame-support, frame-system -* FRAME Pallets -[source, shell] -pallet-assets, pallet-balances, pallet-consensus, pallet-contracts, pallet-council, pallet-democracy, pallet-example, -frame-executive, pallet-session, pallet-staking, pallet-timestamp, pallet-treasury -* Node -[source, shell] -node-cli, node-consensus, node-executor, node-network, node-primitives, kitchensink-runtime -* Subkey -[source, shell] -subkey - -=== Contributing to documentation for Substrate packages - -https://doc.rust-lang.org/1.9.0/book/documentation.html[Document source code] for Substrate packages by annotating the source code with documentation comments. - -Example (generic): -```markdown -/// Summary -/// -/// Description -/// -/// # Panics -/// -/// # Errors -/// -/// # Safety -/// -/// # Examples -/// -/// Summary of Example 1 -/// -/// ```rust -/// // insert example 1 code here -/// ``` -/// -``` - -* Important notes: -** Documentation comments must use annotations with a triple slash `///` -** Modules are documented using `//!` -``` -//! Summary (of module) -//! -//! Description (of module) -``` -* Special section header is indicated with a hash `#`. -** `Panics` section requires an explanation if the function triggers a panic -** `Errors` section is for describing conditions under which a function of method returns `Err(E)` if it returns a `Result` -** `Safety` section requires an explanation if the function is `unsafe` -** `Examples` section includes examples of using the function or method -* Code block annotations for examples are included between triple graves, as shown above. -Instead of including the programming language to use for syntax highlighting as the annotation -after the triple graves, alternative annotations include the `ignore`, `text`, `should_panic`, or `no_run`. -* Summary sentence is a short high level single sentence of its functionality -* Description paragraph is for details additional to the summary sentence -* Missing documentation annotations may be used to identify where to generate warnings with `#![warn(missing_docs)]` -or errors `#![deny(missing_docs)]` -* Hide documentation for items with `#[doc(hidden)]` - -=== Contributing to documentation (tests, extended examples, macros) for Substrate packages - -The code block annotations in the `# Example` section may be used as https://doc.rust-lang.org/1.9.0/book/documentation.html#documentation-as-tests[documentation as tests and for extended examples]. - -* Important notes: -** Rustdoc will automatically add a `main()` wrapper around the code block to test it -** https://doc.rust-lang.org/1.9.0/book/documentation.html#documenting-macros[Documenting macros]. -** Documentation as tests examples are included when running `cargo test` - -== Contributing - -=== Contributing Guidelines - -include::CONTRIBUTING.md[] - -=== Contributor Code of Conduct - -include::CODE_OF_CONDUCT.md[] - -== License - -/~https://github.com/paritytech/substrate/blob/master/LICENSE[LICENSE] diff --git a/substrate/docs/Structure.adoc b/substrate/docs/Structure.adoc deleted file mode 100644 index 6c810a83c51b9..0000000000000 --- a/substrate/docs/Structure.adoc +++ /dev/null @@ -1,121 +0,0 @@ -= Structure -:Author: Substrate developers -:Revision: 0.3.0 -:toc: -:sectnums: - - -== Overview - -Substrate is split into multiple levels with increasing opinion and decreasing flexibility: - -* primitives -* client -* FRAME (formerly `srml`) - -Putting all these components together we have: - -* Integration Tests -* Node -* Node template -* Subkey - -=== Runtime - -* _found in_: `/primitives` -* _crates prefix_: `sp-` -* _constraints_: -** must be `[no_std]` -** crates may not (dev-)depend on crates in other subfolders of this repo - -In the lowest level, Substrate defines primitives, interfaces and traits to implement any on-chain Substrate transition system and its interactions with the outside world. This is the lowest level of abstraction and opinion that everything else builds upon. - -=== Client - -* _found in_: `/client` -* _crates prefix_: `sc-` -* _constraints_: -** crates may not (dev-)depend on any `frame-`-crates - -In the client you can find a set of crates to construct the outer substrate-node, implementing outer runtime interfaces, thus it depends on `runtime`. It provides the outer building blocks like transaction queue, networking layer, database backend, full* and light-client support. - -=== FRAME (formerly `srml`) - -* _found in_: `/frame` -* _crates prefix_: `frame-` and `pallet-` -* _constraints_: -** all crates that go on chain must be `[no_std]` -** must not (dev-)depend on anything in `/client` - -FRAME is a set of modules that implement specific transition functions and features one might want to have in their runtime. - -_Pallets_ are individual modules within _FRAME._ These are containers that host domain-specific logic. They have the `pallet-` prefix. For example, `pallet-staking` contains logic for staking tokens. - -There are a few crates with the `frame-` prefix. These do not contain domain-specific logic. Rather, they are the main FRAME support infrastructure. These are: - -- Executive -- Metadata -- Support -- System -- Utility - -=== Integration Tests - -* _found in_: `/test` -* _crates prefix_: `substrate-test` -* _constraints_: -** only helpers may be published -** purely testing crates must be `publish = false` - -All tests that have to pull (dev)-dependencies out of their subtree and would thus break the dependency rules are considered integration tests and should be stored in here. Only helper-crates in here shall be published, everything else is expected to be non-publish. - -=== Binaries and template - -* _found in_: `/bin` - -We also provide some binaries pulling from the components creating full applications. - -==== Node - -* _found in_: `/bin/node` - -The default (testing) application pulling together our recommended setup of substrate-client with a wasm-contracts-supporting frame-runtime. The node pulls it all together, constructs the (upgradable) runtime, and wires up the client around it. You can find an example client, which includes a full wasm-contracts chain in `node`. This is also what is being built and run if you do `cargo run`. - -==== Node Template - -* _found in_: `/bin/node-template` - -We also provide a template to get you started building your own node. - -==== Utils - -* _found in_: `/bin/utils` - -- **subkey** - Subkey is a client library to generate keys and sign transactions to send to a substrate node. -- **chain-spec-builder** - The chain spec builder builds a chain specification that includes a Substrate runtime compiled as WASM. To ensure proper functioning of the included runtime compile (or run) the chain spec builder binary in `--release` mode. - -== Internal Dependency Tree - -[ditaa] -.... -+---------------+ +----------------+ -| | | | -| runtime +<------+ frame | -| | | | -+------+-----+--+ +-------------+--+ - ^ ^ ^ - | +----------------+ | - | | | -+------+--------+ | | -| | | | -| client | +--+-------+--------+ -| +<---------+ | -+---------------+ | | - | test /bin/* | - | | - | | - +-------------------+ - -....