Update operator docs (Joystream#281)

* docs: 📝 improve operator example file configs * docs: 📝 email notifications guide improvement * fix: 📝 documentation .env archive endpoint variable * fix: 📝 example env gateway_port:8000 * docs: rework deploy orion documentation * docs: 📝 finalize deployment docs * docs: 📝 minor improvements in the documentations
malchililj · Dec 19, 2023 · 66b990a · 66b990a
1 parent 654b2f5
commit 66b990a
Show file tree

Hide file tree

Showing 6 changed files with 165 additions and 24 deletions.
diff --git a/README.md b/README.md
@@ -1,7 +1,8 @@
-# Orion
-
-Orion is a backed node powering [Atlas](/~https://github.com/Joystream/atlas).
-The project is using [Subsquid framework](https://docs.subsquid.io/) and is based on [squid-substrate-template](/~https://github.com/subsquid/squid-substrate-template).
-
+## Intro
+Orion is the backend powering [Atlas](/~https://github.com/Joystream/atlas)
+## Quickstart
+In order to deploy orion for the first time follow [these instructions](docs/operator-guide/tutorials/deploying-orion.md)
 ## [Developer guide](docs/developer-guide/index.md)
-## [Operator guide](docs/operator-guide.md)
+## [Operator guide](docs/operator-guide/index.md)
+## Support
+In case you are facing issues even after several attempts you can ask for support in the the [DAO app discord channel](https://discord.gg/joystream-811216481340751934) by providing a *detailed description* of your problem 
diff --git a/docs/operator-guide/examples/.env b/docs/operator-guide/examples/.env
@@ -6,14 +6,25 @@ DB_PORT=5432
 DB_ADMIN_USER=admin
 DB_ADMIN_PASS=admin
 
+# ==================================================================================
+# ============================= ARCHIVE CONFIGURATION ==============================
+# ==================================================================================
+# postgres database port used by the archive
+ARCHIVE_DB_PORT=12345
+# Port that is used by the processor to fetch batches of block
+GATEWAY_PORT=8000
+# alternatively you can point to your own joystream rpc node if you have one
+WS_SOURCE=wss://rpc.joystream.org:9944 
+# port exposed by the explorer
+EXPLORER_PORT=4444
+# Archive gateway endpoint
+ARCHIVE_GATEWAY_URL=http://orion_archive_gateway:${GATEWAY_PORT}/graphql
+
 # Orion services ports
 GRAPHQL_API_PORT=4350
 AUTH_API_PORT=4074
 PROCESSOR_PROMETHEUS_PORT=3337
 
-# Archive gateway endpoint
-ARCHIVE_GATEWAY_URL=https://joystream.archive.subsquid.io/graphql
-
 # Express.js "trust proxy" setting
 # It specifies which IP addresses are considered "trusted" to provide
 # correct values in the `X-Forwarded-For` header (to avoid ip spoofing).
@@ -46,22 +57,28 @@ DEV_DISABLE_SAME_SITE=true
 
 # The name of Your gateway (pick a unique one!)
 APP_NAME=<YOUR GATEWAY NAME>
+
 # Whether videos with no category assigned should be visible on the gateway
 SUPPORT_NO_CATEGORY_VIDEOS=<true/false>
+
 # Whether the gateway should support all newly created video categories
 # by default.
 SUPPORT_NEW_CATEGORIES=<true/false>
+
 # Default value of the "kill switch". If set to true, it will force the
 # Atlas to start in a "maintenance mode", before the value of the kill
 # switch is changed via the operator API.
 KILL_SWITCH_ON=<true/false>
+
 # The number of seconds that need to pass before a video view from the
 # same user is counted again.
 VIDEO_VIEW_PER_USER_TIME_LIMIT=300
+
 # A secret string that will be used when authenticating as a Gateway operator.
 # It's recommended to generate a random string of at least 16 characters.
 # For example: `openssl rand -base64 18`
 OPERATOR_SECRET=<SECURELY RANDOM SECRET>
+
 # Weights used for calclating the relevance score of a video.
 # The relevance score affects the order in which videos are displayed
 # on the homepage of Atlas.
@@ -72,12 +89,15 @@ OPERATOR_SECRET=<SECURELY RANDOM SECRET>
 # The fifth value is a list of two values:
 #   - joystreamCreationWeight: how much Video.createdAt date affects the `newness` score
 #   - ytCreationWeight: how much Video.publishedBeforeJoystream affects the `newness` score
+# The sixth value is the "channel_weight".
 # The relevance score is calculated as follows:
 # ```
+# (
 # newness * newness_weight +
 # views * views_weight +
 # comments * comments_weight +
 # reactions * reactions_weight
+# ) * channel_weight
 # ```
 # Where newness is:
 # ```
@@ -86,42 +106,53 @@ OPERATOR_SECRET=<SECURELY RANDOM SECRET>
 #   (days since Video.publishedBeforeJoystream * ytCreationWeight)
 # ) / (joystreamCreationWeight + ytCreationWeight)
 # ```
-RELEVANCE_WEIGHTS="[1, 0.03, 0.3, 0.5, [7,3]]"
+RELEVANCE_WEIGHTS="[1, 0.03, 0.3, 0.5, [7,3], 1]"
+
 # After how many new video views, the relevance score of a video should
 # be recalculated.
 VIDEO_RELEVANCE_VIEWS_TICK=10
+
 # Maximum number of entities that can be cached in Orion processor's memory.
 # If the Orion processor service is running out of memory, try decreasing
 # this value.
 MAX_CACHED_ENTITIES=1000
+
 # A secret string used to generate a private key for signing the attestation
 # that certain channels and videos were created throguh Your gateway.
 # It's recommended to generate a random string of at least 16 characters.
 # For example: `openssl rand -base64 18`
 APP_PRIVATE_KEY=<SECURELY RANDOM SECRET>
+
 # After how many minutes does the session expire in case of user inactivity.
 # Once the session expires, the user will need to re-authenticate.
 SESSION_EXPIRY_AFTER_INACTIVITY_MINUTES=60
+
 # After how many hours does the session expire regardless of user activity.
 SESSION_MAX_DURATION_HOURS=720
+
 # The route in Atlas used for the purpose of user e-mail confirmation.
 # CURRENTLY NOT USED
 EMAIL_CONFIRMATION_ROUTE=NOT_USED
+
 # The number of hours after which the e-mail confirmation link expires.
 EMAIL_CONFIRMATION_TOKEN_EXPIRY_TIME_HOURS=24
+
 # How many e-mail confirmation links can be sent to a user within
 # `EMAIL_CONFIRMATION_TOKEN_EXPIRY_TIME_HOURS` (to prevent spamming)
 EMAIL_CONFIRMATION_TOKEN_RATE_LIMIT=5
+
 # How many seconds have to pass since the timestamp included in a signed message
 # that proves the ownership of a Joystream account/address, before the message
 # is considered expired.
 ACCOUNT_OWNERSHIP_PROOF_EXPIRY_TIME_SECONDS=300 # 5 minutes
+
 # A secret string used to generate a private key for the purpose of signing cookies,
 # to ensure that the cookies sent by the client application (like Atlas) have been
 # indeed set by your instance of Orion and have not been tampered with.
 # It's recommended to generate a random string of at least 16 characters.
 # For example: `openssl rand -base64 18`
 COOKIE_SECRET=<SECURELY RANDOM SECRET>
+
 # max number of attempts to deliver email notification
 EMAIL_NOTIFICATION_DELIVERY_MAX_ATTEMPTS=5
 
@@ -133,5 +164,6 @@ SENDGRID_FROM_EMAIL=<NO-REPLY EMAIL ADDRESS>
 # Notification emails storage links
 # Online path which contains the icons and logo to display in the notification e-mails
 APP_ASSET_STORAGE=<LINK_TO_STORAGE_ASSETS>
+
 # Name of the application link displayed in the notification e-mail footer
 APP_NAME_ALT=<ALTERNATIVE_APP_NAME>
diff --git a/docs/operator-guide/examples/Caddyfile b/docs/operator-guide/examples/Caddyfile
@@ -6,7 +6,12 @@ query.{$GATEWAY_ROOT_DOMAIN} {
     reverse_proxy orion-graphql-api:{$GRAPHQL_API_PORT}
 }
 
-{$GATEWAY_ROOT_DOMAIN} {
-    root * /srv/atlas
-    file_server
-}
+archive.{$GATEWAY_ROOT_DOMAIN} {
+    reverse_proxy orion-graphql-api:{$GATEWAY_PORT}
+}
+
+# # Optional: uncomment to serve atlas statically on the server
+# {$GATEWAY_ROOT_DOMAIN} {
+#     root * /srv/atlas
+#     file_server
+# }
diff --git a/docs/operator-guide/examples/docker-compose.yml b/docs/operator-guide/examples/docker-compose.yml
@@ -33,7 +33,7 @@ services:
       - ./db/export:/squid/db/export
     command: >
       sh -c "npx squid-typeorm-migration apply && \
-             node lib/processor.js"
+            node lib/processor.js"
     ports:
       - '127.0.0.1:${PROCESSOR_PROMETHEUS_PORT}:${PROCESSOR_PROMETHEUS_PORT}'
   # Orion GraphQL API
@@ -84,6 +84,78 @@ services:
       - ./caddy/data:/data
       - ./caddy/config:/config
 
+  archive_db:
+    container_name: orion_archive_db
+    hostname: orion_archive_db
+    image: postgres:14 # CockroachDB cluster might be a better fit for production deployment
+    restart: unless-stopped
+    volumes:
+      - orion_archive_db_data:/var/lib/postgresql/data
+    environment:
+      POSTGRES_USER: postgres
+      POSTGRES_PASSWORD: postgres
+      POSTGRES_DB: squid-archive
+    ports:
+      - '127.0.0.1:${ARCHIVE_DB_PORT}:${ARCHIVE_DB_PORT}'
+      - '[::1]:${ARCHIVE_DB_PORT}:${ARCHIVE_DB_PORT}'
+    command: ['postgres', '-c', 'log_statement=all', '-p', '${ARCHIVE_DB_PORT}']
+
+  ingest:
+    container_name: orion_archive_ingest
+    depends_on:
+      - archive_db
+    restart: unless-stopped
+    image: subsquid/substrate-ingest:firesquid
+    command: [
+        '-e',
+        '${WS_SOURCE}',
+        '-c',
+        '20', # allow up to 20 pending requests for the above endpoint (default is 5)
+        #  "--start-block", "1000000", # uncomment to specify a non-zero start block
+        '--out',
+        'postgres://postgres:postgres@orion_archive_db:${ARCHIVE_DB_PORT}/squid-archive',
+      ]
+
+  gateway:
+    container_name: orion_archive_gateway
+    hostname: orion_archive_gateway
+    depends_on:
+      - archive_db
+    restart: unless-stopped
+    image: subsquid/substrate-gateway:firesquid
+    environment:
+      DATABASE_MAX_CONNECTIONS: 5
+      RUST_LOG: 'actix_web=info,actix_server=info'
+    command: [
+        '--database-url',
+        'postgres://postgres:postgres@orion_archive_db:${ARCHIVE_DB_PORT}/squid-archive',
+        # "--evm-support" # uncomment for chains with Frontier EVM pallet
+        # (e.g. Moonbeam/Moonriver or Astar/Shiden)
+      ]
+    ports:
+      - '127.0.0.1:${GATEWAY_PORT}:8000'
+      - '[::1]:${GATEWAY_PORT}:8000'
+
+  # Explorer service is optional.
+  # It provides rich GraphQL API for querying archived data.
+  # Many developers find it very useful for exploration and debugging.
+  explorer:
+    container_name: orion_archive_explorer
+    hostname: orion_archive_explorer
+    restart: unless-stopped
+    image: subsquid/substrate-explorer:firesquid
+    environment:
+      DB_TYPE: postgres # set to `cockroach` for Cockroach DB
+      DB_HOST: archive_db
+      DB_PORT: '12345'
+      DB_NAME: 'squid-archive'
+      DB_USER: 'postgres'
+      DB_PASS: 'postgres'
+    ports:
+      - '${EXPLORER_PORT}:3000'
+
 volumes:
+  orion_archive_db_data:
+    driver: local
   orion-db-data:
     driver: local
diff --git a/docs/operator-guide/tutorials/deploying-orion.md b/docs/operator-guide/tutorials/deploying-orion.md
@@ -4,6 +4,8 @@ When deploying Orion to production, you should host it under the same root domai
 
 There are multiple reverse proxy servers that you can use to host Orion in production, like [Nginx](https://www.nginx.com/) or [Caddy](https://caddyserver.com/). In this guide we'll be focusing on Caddy because of its simplicity.
 
+The following guide will allow you to deploy the **latest** version of Orion
+
 ## Prerequisites
 
 On your server, you'll need to have [Docker](https://docs.docker.com/) and `docker-compose` installed (see: [installation instructions](https://docs.docker.com/engine/install/))
@@ -17,13 +19,18 @@ You will also need a sendgrid account with an api key in order to have email not
 
 ## Step-by-step guide
 
+### 1. Orion setup
 1. Copy the `.env`, `docker-compose.yml`, `postgres.conf` and `Caddyfile` from the [`examples`](../examples/) directory to your server.
 1. Configure the Orion production environment variables. Inside the `.env` file you will find comments describing the meaning of each of the variables. Most importantly make sure to:
     1. Replace all "placeholder" values inside angle brackets (`<>`) with the real values
     1. Fill in all the secrets (`OPERATOR_SECRET`, `APP_PRIVATE_KEY`, `COOKIE_SECRET`) with proper, randomly generated values
     1. **Make sure that `ORION_ENV` is set to `production`!**
     1. Set `GATEWAY_ROOT_DOMAIN` to your Gateway's root domain, in our example case it'll be `mygateway.com`
-    1. Configure the email notification scheduler chron job in order to have notification delivery via email (see: [Email Notifications](./email-notifications.md))
+    1. (Optional) Configure the email notification scheduler chron job (for Orion >= `3.2.0`) in order to have notification delivery via email (see: [Email Notifications](./email-notifications.md))
+
+### 2. Atlas static Deployment (optional)
+This step is optional in case you are considering using a different deployment service like Vercel for example
+
 1. Prepare a production build of the Atlas app. Assuming you have already cloned the Atlas repository and configured the environment variables inside `packages/atlas/src/.env` as described in the [Atlas operator guide](/~https://github.com/Joystream/atlas/blob/master/docs/operator-guide.md), you'll also need to:
     1. Make sure the `VITE_ENV` value is set to `production`
     1. Set `VITE_PRODUCTION_ORION_AUTH_URL` to your Orion Auth API endpoint (`https://auth.mygateway.com/api/v1` in our example case)
@@ -56,10 +63,17 @@ You will also need a sendgrid account with an api key in order to have email not
     ├── docker-compose.yml
     └── postgres.conf
     ```
-1. Once everything is set up, you can start the Orion services and the Caddy server by running:
-    ```bash
-    docker-compose up -d --profile deploy
-    ```
 
 That's it! Your gateway should now be available under `https://mygateway.com`!
-If you kept the Auth API playground enabled in your `.env` (`OPENAPI_PLAYGROUND=true`), you can also use `https://auth.mygateway.com/playground` to perform the Operator authentication (as described in _[Local testing](./local-testing.md#authentication)_ tutorial) and then execute Operator queries and mutations through `https://query.mygateway.com/graphql`.
+If you kept the Auth API playground enabled in your `.env` (`OPENAPI_PLAYGROUND=true`), you can also use `https://auth.mygateway.com/playground` to perform the Operator authentication (as described in _[Local testing](./local-testing.md#authentication)_ tutorial) and then execute Operator queries and mutations through `https://query.mygateway.com/graphql`.
+
+### 3. Orion deployment
+Once everything is set up, you can start:
+a.The Orion services only with:
+    ```bash
+    docker-compose up -d
+    ```
+b.The Orion services and the Caddy server by running:
+    ```bash
+    docker-compose --profile deploy up -d
+    ```
diff --git a/docs/operator-guide/tutorials/email-notifications.md b/docs/operator-guide/tutorials/email-notifications.md
@@ -1,6 +1,18 @@
 # Operator email notifications guide
 
-This feature has been introduced in Orion starting from version 3.2.0
+This feature has been introduced in Orion starting from version 3.2.0 and the purpose of this documentation is 
+to instruct a potential gateway operator about how setting up a email notification system using a chron scheduler
+
+## Prerequisite
+Currently in order to have the scheduling working you must clone the orion repo locally via:
+```sh
+    git clone /~https://github.com/Joystream/orion.git
+```
+and cd-ing into it and installing the dependencies and building using (which assumes you have [npm](https://www.npmjs.com/) installed)
+```sh
+    cd orion
+    make prepare
+```
 
 The current implementation uses a chron job for executing the
 `src/lib/mail-scheduler/index.js` script at every specified interval.
@@ -18,9 +30,14 @@ Configure the `.env` variables with:
 1. Open up the chrontab file with `crontab -e`
 2. Add the following line:
 ```bash
-*/<TIME_INTERVAL_MINUTES> * * * * /home/ubuntu/.volta/bin/node <PATH_TO_ORION_FOLDER>/lib/mail-scheduler/index.js >> <PATH_TO_ORION_FOLDER>/chron_mail_scheduler.log 2>&1
+*/<TIME_INTERVAL_MINUTES> * * * * <NODE_BINARY_PATH> /lib/mail-scheduler/index.js >> <PATH_TO_ORION_FOLDER>/chron_mail_scheduler.log 2>&1
 ```
-replacing the `<PATH_TO_ORION_FOLDER>` with the path to the orion directory cloned for the setup and `<TIME_INTERVAL_MINUTES>` with the desired time interval in minutes. This will make the script to be executed with the desired frequency and it will output the log into a `<PATH_TO_ORION_FOLDER>/chron_mail_scheduler.log` file
+replacing:
+    - `<PATH_TO_ORION_FOLDER>` with the absolute path to the orion directory cloned in the *prerequise* step
+    - `<TIME_INTERVAL_MINUTES>` with the desired time interval in minutes.
+    - `<NODE_BINARY_PATH>`with the absolute path for the node-js binary (which can be obtained by running the shell line `which node`)
+
+This will make the script to be executed with the desired frequency and it will output the log into a `<PATH_TO_ORION_FOLDER>/chron_mail_scheduler.log` file
 3. save and close the crontab file 
 
 ### Updating the interval value