Merge branch 'develop' into hughns/msc3882-r1

matrix-org · May 30, 2023 · 6f26ca4 · 6f26ca4
2 parents 210c22d + 6fc3deb
commit 6f26ca4
Show file tree

Hide file tree

Showing 111 changed files with 3,741 additions and 999 deletions.
diff --git a/.github/workflows/dependabot_changelog.yml b/.github/workflows/dependabot_changelog.yml
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
@@ -314,7 +314,7 @@ jobs:
       # There aren't wheels for some of the older deps, so we need to install
       # their build dependencies
       - run: |
-          sudo apt update
+          sudo apt-get -qq update
           sudo apt-get -qq install build-essential libffi-dev python-dev \
             libxml2-dev libxslt-dev xmlsec1 zlib1g-dev libjpeg-dev libwebp-dev
 

diff --git a/CHANGES.md b/CHANGES.md
@@ -1,3 +1,97 @@
+Synapse 1.85.0rc1 (2023-05-30)
+==============================
+
+Features
+--------
+
+- Improve performance of backfill requests by performing backfill of previously failed requests in the background. ([\#15585](/~https://github.com/matrix-org/synapse/issues/15585))
+- Add a new [admin API](https://matrix-org.github.io/synapse/v1.85/usage/administration/admin_api/index.html) to [create a new device for a user](https://matrix-org.github.io/synapse/v1.85/admin_api/user_admin_api.html#create-a-device). ([\#15611](/~https://github.com/matrix-org/synapse/issues/15611))
+- Add Unix socket support for Redis connections. Contributed by Jason Little. ([\#15644](/~https://github.com/matrix-org/synapse/issues/15644))
+
+
+Bugfixes
+--------
+
+- Fix a long-standing bug where setting the read marker could fail when using message retention. Contributed by Nick @ Beeper (@fizzadar). ([\#15464](/~https://github.com/matrix-org/synapse/issues/15464))
+- Fix a long-standing bug where the `url_preview_url_blacklist` configuration setting was not applied to oEmbed or image URLs found while previewing a URL. ([\#15601](/~https://github.com/matrix-org/synapse/issues/15601))
+- Fix a long-standing bug where filters with multiple backslashes were rejected. ([\#15607](/~https://github.com/matrix-org/synapse/issues/15607))
+- Fix a bug introduced in Synapse 1.82.0 where the error message displayed when validation of the `app_service_config_files` config option fails would be incorrectly formatted. ([\#15614](/~https://github.com/matrix-org/synapse/issues/15614))
+- Fix a long-standing bug where deactivated users were still able to login using the custom `org.matrix.login.jwt` login type (if enabled). ([\#15624](/~https://github.com/matrix-org/synapse/issues/15624))
+- Fix a long-standing bug where deactivated users were able to login in uncommon situations. ([\#15634](/~https://github.com/matrix-org/synapse/issues/15634))
+
+
+Improved Documentation
+----------------------
+
+- Warn users that at least 3.75GB of space is needed for the nix Synapse development environment. ([\#15613](/~https://github.com/matrix-org/synapse/issues/15613))
+- Remove outdated comment from the generated and sample homeserver log configs. ([\#15648](/~https://github.com/matrix-org/synapse/issues/15648))
+- Improve contributor docs to make it more clear that Rust is a necessary prerequisite. Contributed by @grantm. ([\#15668](/~https://github.com/matrix-org/synapse/issues/15668))
+
+
+Deprecations and Removals
+-------------------------
+
+- Remove the old version of the R30 (30-day retained users) phone-home metric. ([\#10428](/~https://github.com/matrix-org/synapse/issues/10428))
+
+
+Internal Changes
+----------------
+
+- Create dependabot changelogs at release time. ([\#15481](/~https://github.com/matrix-org/synapse/issues/15481))
+- Add not null constraint to column `full_user_id` of tables `profiles` and `user_filters`. ([\#15537](/~https://github.com/matrix-org/synapse/issues/15537))
+- Allow connecting to HTTP Replication Endpoints by using `worker_name` when constructing the request. ([\#15578](/~https://github.com/matrix-org/synapse/issues/15578))
+- Make the `thread_id` column on `event_push_actions`, `event_push_actions_staging`, and `event_push_summary` non-null. ([\#15597](/~https://github.com/matrix-org/synapse/issues/15597))
+- Run mypy type checking with the minimum supported Python version to catch new usage that isn't backwards-compatible. ([\#15602](/~https://github.com/matrix-org/synapse/issues/15602))
+- Fix subscriptable type usage in Python <3.9. ([\#15604](/~https://github.com/matrix-org/synapse/issues/15604))
+- Update internal terminology. ([\#15606](/~https://github.com/matrix-org/synapse/issues/15606), [\#15620](/~https://github.com/matrix-org/synapse/issues/15620))
+- Instrument `state` and `state_group` storage-related operations to better picture what's happening when tracing. ([\#15610](/~https://github.com/matrix-org/synapse/issues/15610), [\#15647](/~https://github.com/matrix-org/synapse/issues/15647))
+- Trace how many new events from the backfill response we need to process. ([\#15633](/~https://github.com/matrix-org/synapse/issues/15633))
+- Re-type config paths in `ConfigError`s to be `StrSequence`s instead of `Iterable[str]`s. ([\#15615](/~https://github.com/matrix-org/synapse/issues/15615))
+- Update Mutual Rooms ([MSC2666](/~https://github.com/matrix-org/matrix-spec-proposals/pull/2666)) implementation to match new proposal text. ([\#15621](/~https://github.com/matrix-org/synapse/issues/15621))
+- Remove the unstable identifiers from faster joins ([MSC3706](/~https://github.com/matrix-org/matrix-spec-proposals/pull/3706)). ([\#15625](/~https://github.com/matrix-org/synapse/issues/15625))
+- Fix the olddeps CI. ([\#15626](/~https://github.com/matrix-org/synapse/issues/15626))
+- Remove duplicate timestamp from test logs (`_trial_temp/test.log`). ([\#15636](/~https://github.com/matrix-org/synapse/issues/15636))
+- Fix two memory leaks in `trial` test runs. ([\#15630](/~https://github.com/matrix-org/synapse/issues/15630))
+- Limit the size of the `HomeServerConfig` cache in trial test runs. ([\#15646](/~https://github.com/matrix-org/synapse/issues/15646))
+- Improve type hints. ([\#15658](/~https://github.com/matrix-org/synapse/issues/15658), [\#15659](/~https://github.com/matrix-org/synapse/issues/15659))
+- Add requesting user id parameter to key claim methods in `TransportLayerClient`. ([\#15663](/~https://github.com/matrix-org/synapse/issues/15663))
+- Speed up rebuilding of the user directory for local users. ([\#15665](/~https://github.com/matrix-org/synapse/issues/15665))
+- Implement "option 2" for [MSC3820](/~https://github.com/matrix-org/matrix-spec-proposals/pull/3820): Room version 11. ([\#15666](/~https://github.com/matrix-org/synapse/issues/15666), [\#15678](/~https://github.com/matrix-org/synapse/issues/15678))
+
+### Updates to locked dependencies
+
+* Bump furo from 2023.3.27 to 2023.5.20. ([\#15642](/~https://github.com/matrix-org/synapse/issues/15642))
+* Bump log from 0.4.17 to 0.4.18. ([\#15681](/~https://github.com/matrix-org/synapse/issues/15681))
+* Bump prometheus-client from 0.16.0 to 0.17.0. ([\#15682](/~https://github.com/matrix-org/synapse/issues/15682))
+* Bump pydantic from 1.10.7 to 1.10.8. ([\#15685](/~https://github.com/matrix-org/synapse/issues/15685))
+* Bump pygithub from 1.58.1 to 1.58.2. ([\#15643](/~https://github.com/matrix-org/synapse/issues/15643))
+* Bump requests from 2.28.2 to 2.31.0. ([\#15651](/~https://github.com/matrix-org/synapse/issues/15651))
+* Bump sphinx from 6.1.3 to 6.2.1. ([\#15641](/~https://github.com/matrix-org/synapse/issues/15641))
+* Bump types-bleach from 6.0.0.1 to 6.0.0.3. ([\#15686](/~https://github.com/matrix-org/synapse/issues/15686))
+* Bump types-pillow from 9.5.0.2 to 9.5.0.4. ([\#15640](/~https://github.com/matrix-org/synapse/issues/15640))
+* Bump types-pyyaml from 6.0.12.9 to 6.0.12.10. ([\#15683](/~https://github.com/matrix-org/synapse/issues/15683))
+* Bump types-requests from 2.30.0.0 to 2.31.0.0. ([\#15684](/~https://github.com/matrix-org/synapse/issues/15684))
+* Bump types-setuptools from 67.7.0.2 to 67.8.0.0. ([\#15639](/~https://github.com/matrix-org/synapse/issues/15639))
+
+Synapse 1.84.1 (2023-05-26)
+===========================
+
+This patch release fixes a major issue with homeservers that do not have an `instance_map` defined but which do use workers.
+If you have already upgraded to Synapse 1.84.0 and your homeserver is working normally, then there is no need to update to this patch release.
+
+
+Bugfixes
+--------
+
+- Fix a bug introduced in Synapse v1.84.0 where workers do not start up when no `instance_map` was provided. ([\#15672](/~https://github.com/matrix-org/synapse/issues/15672))
+
+
+Internal Changes
+----------------
+
+- Add `dch` and `notify-send` to the development Nix flake so that the release script can be used. ([\#15673](/~https://github.com/matrix-org/synapse/issues/15673))
+
+
 Synapse 1.84.0 (2023-05-23)
 ===========================
 

diff --git a/Cargo.lock b/Cargo.lock
diff --git a/changelog.d/10428.removal b/changelog.d/10428.removal
diff --git a/changelog.d/15464.bugfix b/changelog.d/15464.bugfix
diff --git a/changelog.d/15537.misc b/changelog.d/15537.misc
diff --git a/changelog.d/15578.misc b/changelog.d/15578.misc
diff --git a/changelog.d/15582.feature b/changelog.d/15582.feature
@@ -0,0 +1 @@
+Experimental [MSC3861](/~https://github.com/matrix-org/matrix-spec-proposals/pull/3861) support: delegate auth to an OIDC provider.
diff --git a/changelog.d/15599.bugfix b/changelog.d/15599.bugfix
diff --git a/changelog.d/15601.bugfix b/changelog.d/15601.bugfix
diff --git a/changelog.d/15602.misc b/changelog.d/15602.misc
diff --git a/changelog.d/15604.misc b/changelog.d/15604.misc
diff --git a/changelog.d/15606.misc b/changelog.d/15606.misc
diff --git a/changelog.d/15607.bugfix b/changelog.d/15607.bugfix
diff --git a/changelog.d/15610.misc b/changelog.d/15610.misc
diff --git a/changelog.d/15611.feature b/changelog.d/15611.feature
diff --git a/changelog.d/15613.doc b/changelog.d/15613.doc
diff --git a/changelog.d/15614.bugfix b/changelog.d/15614.bugfix
diff --git a/changelog.d/15615.misc b/changelog.d/15615.misc
diff --git a/changelog.d/15620.misc b/changelog.d/15620.misc
diff --git a/changelog.d/15621.misc b/changelog.d/15621.misc
diff --git a/changelog.d/15624.bugfix b/changelog.d/15624.bugfix
diff --git a/changelog.d/15625.misc b/changelog.d/15625.misc
diff --git a/changelog.d/15626.misc b/changelog.d/15626.misc
diff --git a/changelog.d/15630.misc b/changelog.d/15630.misc
diff --git a/changelog.d/15633.misc b/changelog.d/15633.misc
diff --git a/changelog.d/15634.bugfix b/changelog.d/15634.bugfix
diff --git a/changelog.d/15636.misc b/changelog.d/15636.misc
diff --git a/changelog.d/15639.misc b/changelog.d/15639.misc
diff --git a/changelog.d/15640.misc b/changelog.d/15640.misc
diff --git a/changelog.d/15641.misc b/changelog.d/15641.misc
diff --git a/changelog.d/15642.misc b/changelog.d/15642.misc
diff --git a/changelog.d/15643.misc b/changelog.d/15643.misc
diff --git a/changelog.d/15646.misc b/changelog.d/15646.misc
diff --git a/changelog.d/15647.bugfix b/changelog.d/15647.bugfix
diff --git a/changelog.d/15648.doc b/changelog.d/15648.doc
diff --git a/changelog.d/15651.misc b/changelog.d/15651.misc
diff --git a/changelog.d/15658.misc b/changelog.d/15658.misc
diff --git a/changelog.d/15690.misc b/changelog.d/15690.misc
@@ -0,0 +1 @@
+Remove some unused code.
diff --git a/debian/changelog b/debian/changelog
@@ -1,3 +1,15 @@
+matrix-synapse-py3 (1.85.0~rc1) stable; urgency=medium
+
+  * New Synapse release 1.85.0rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 30 May 2023 13:56:54 +0100
+
+matrix-synapse-py3 (1.84.1) stable; urgency=medium
+
+  * New Synapse release 1.84.1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Fri, 26 May 2023 16:15:30 +0100
+
 matrix-synapse-py3 (1.84.0) stable; urgency=medium
 
   * New Synapse release 1.84.0.

diff --git a/docs/development/contributing_guide.md b/docs/development/contributing_guide.md
@@ -22,6 +22,9 @@ on Windows is not officially supported.
 
 The code of Synapse is written in Python 3. To do pretty much anything, you'll need [a recent version of Python 3](https://www.python.org/downloads/). Your Python also needs support for [virtual environments](https://docs.python.org/3/library/venv.html). This is usually built-in, but some Linux distributions like Debian and Ubuntu split it out into its own package. Running `sudo apt install python3-venv` should be enough.
 
+A recent version of the Rust compiler is needed to build the native modules. The
+easiest way of installing the latest version is to use [rustup](https://rustup.rs/).
+
 Synapse can connect to PostgreSQL via the [psycopg2](https://pypi.org/project/psycopg2/) Python library. Building this library from source requires access to PostgreSQL's C header files. On Debian or Ubuntu Linux, these can be installed with `sudo apt install libpq-dev`.
 
 Synapse has an optional, improved user search with better Unicode support. For that you need the development package of `libicu`. On Debian or Ubuntu Linux, this can be installed with `sudo apt install libicu-dev`.
@@ -30,9 +33,6 @@ The source code of Synapse is hosted on GitHub. You will also need [a recent ver
 
 For some tests, you will need [a recent version of Docker](https://docs.docker.com/get-docker/).
 
-A recent version of the Rust compiler is needed to build the native modules. The
-easiest way of installing the latest version is to use [rustup](https://rustup.rs/).
-
 
 # 3. Get the source.
 
@@ -53,6 +53,11 @@ can find many good git tutorials on the web.
 
 # 4. Install the dependencies
 
+
+Before installing the Python dependencies, make sure you have installed a recent version
+of Rust (see the "What do I need?" section above). The easiest way of installing the
+latest version is to use [rustup](https://rustup.rs/).
+
 Synapse uses the [poetry](https://python-poetry.org/) project to manage its dependencies
 and development environment. Once you have installed Python 3 and added the
 source, you should install `poetry`.
@@ -76,22 +81,40 @@ cd path/where/you/have/cloned/the/repository
 poetry install --extras all
 ```
 
-This will install the runtime and developer dependencies for the project.
+This will install the runtime and developer dependencies for the project.  Be sure to check
+that the `poetry install` step completed cleanly.
 
 ## Running Synapse via poetry
 
 To start a local instance of Synapse in the locked poetry environment, create a config file:
 
 ```sh
 cp docs/sample_config.yaml homeserver.yaml
+cp docs/sample_log_config.yaml log_config.yaml
 ```
 
-Now edit homeserver.yaml, and run Synapse with:
+Now edit `homeserver.yaml`, things you might want to change include:
+
+- Set a `server_name`
+- Adjusting paths to be correct for your system like the `log_config` to point to the log config you just copied
+- Using a [PostgreSQL database instead of SQLite](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#database)
+- Adding a [`registration_shared_secret`](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#registration_shared_secret) so you can use [`register_new_matrix_user` command](https://matrix-org.github.io/synapse/latest/setup/installation.html#registering-a-user).
+
+And then run Synapse with the following command:
 
 ```sh
 poetry run python -m synapse.app.homeserver -c homeserver.yaml
 ```
 
+If you get an error like the following:
+
+```
+importlib.metadata.PackageNotFoundError: matrix-synapse
+```
+
+this probably indicates that the `poetry install` step did not complete cleanly - go back and
+resolve any issues and re-run until successful.
+
 # 5. Get in touch.
 
 Join our developer community on Matrix: [#synapse-dev:matrix.org](https://matrix.to/#/#synapse-dev:matrix.org)!

diff --git a/docs/development/dependencies.md b/docs/development/dependencies.md
@@ -260,15 +260,17 @@ doesn't require poetry. (It's what we use in CI too). However, you could try
 
 ## ...handle a Dependabot pull request?
 
-Synapse uses Dependabot to keep the `poetry.lock` file up-to-date. When it
-creates a pull request a GitHub Action will run to automatically create a changelog
-file. Ensure that:
+Synapse uses Dependabot to keep the `poetry.lock` and `Cargo.lock` file 
+up-to-date with the latest releases of our dependencies. The changelog check is
+omitted for Dependabot PRs; the release script will include them in the 
+changelog.
+
+When reviewing a dependabot PR, ensure that:
 
 * the lockfile changes look reasonable;
 * the upstream changelog file (linked in the description) doesn't include any
   breaking changes;
-* continuous integration passes (due to permissions, the GitHub Actions run on
-  the changelog commit will fail, look at the initial commit of the pull request);
+* continuous integration passes.
 
 In particular, any updates to the type hints (usually packages which start with `types-`)
 should be safe to merge if linting passes.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Experimental [MSC3861](/~https://github.com/matrix-org/matrix-spec-proposals/pull/3861) support: delegate auth to an OIDC provider.