Failed to get state for roomid after too early kick

[verymilan]

Description

A remote user was invited to a portal room created by the mautrix telegram bridge, causing the bridge kicking them out as soon as they tried to join. They now are in a room with unread messages that they can't configure, leave or purge. Not even force_purge helps. It purges forever until the server is restarted.
When a purge is invoked, an error is thrown in the serverlog: sync_partial_state_room-0 - Failed to get state for roomid at <FrozenEventV3 event_id ... We can't get valid state history., giving up!
Receiving room state via the api returns nothing and runs forever as well.

Steps to reproduce

• login with the /~https://github.com/mautrix/telegram bot
• invite an innocent, not logged into telegram remote matrix user to a matrix mirrored room the bot has created before automatically
• have the user join
• the bot will remove the user throwing This chat does not have a bot relaying messages for unauthenticated users
• the user now won't be able to do anything with the room and has a permanent unread notification

Homeserver

pragma-messenger.ch

Synapse Version

1.76.0

Installation Method

pip (from PyPI)

Database

PostgreSQL

Workers

Multiple workers

Platform

Debian 11, no containers or vms.

Configuration

Presence and Message retention are turned on. The Mjolnir module is installed.

Relevant log output

2023-02-12 09:54:10,821 - synapse.handlers.federation - 1924 - ERROR - sync_partial_state_room-0 - Failed to get state for !BViUDwYyFqaSxzDVuu:tchncs.de at <FrozenEventV3 event_id=$OsWs6l7AXqdA32_PRKAkfX7wk4y-5ojQeY9tfatVzBM, type=m.room.member, state_key=@wuethrich:pragma-messenger.ch, outlier=False> from tchncs.de because ERROR 403: We can't get valid state history., giving up!
2023-02-12 12:22:21,244 - synapse.http.matrixfederationclient - 709 - WARNING - sync_partial_state_room-0-$OsWs6l7AXqdA32_PRKAkfX7wk4y-5ojQeY9tfatVzBM-$npMAIwUs-g4ABVY53p3rK5E8oe8-ptcJ2PzGHjkHprA - {GET-O-1} [tchncs.de] Request failed: GET matrix://tchncs.de/_matrix/federation/v1/state_ids/%21BViUDwYyFqaSxzDVuu%3Atchncs.de?event_id=%24npMAIwUs-g4ABVY53p3rK5E8oe8-ptcJ2PzGHjkHprA: HttpResponseException('403: Forbidden')

Anything else that would be useful to know?

The room was already purged on tchncs.de in an attempt to help the remote server – but with no luck (it's gone on tchncs.de tho). The affected room has room version 6.

[DMRobertson]

Can you see any other log lines for that room ID !BViUDwYyFqaSxzDVuu:tchncs.de?

I would guess that this has hit #14859

[verymilan]

At least not with the current loglevel WARNING... 🤔 (other than some clientside disconnects)

[DMRobertson]

Can you find the logs corresponding to the original purge? (Try grepping for DELETE.*/rooms/)

[verymilan]

Sadly i can only find a syntax error from right before the first purge (where i had an error in the pruge:true json syntax)

[DMRobertson]

#15068 should hopefully make the purge work.

I don't yet understand what happened before the purge. Perhaps we can find something to improve there... will have to think some more on this.

[DMRobertson]

@verymilan If it's possible to wait for #15068 to land in a release, I'd recommend upgrading to it and trying to purge the room again.

If that's not practical and you want to purge this room more urgently, I can provide instructions to manually do database surgery; but I'd prefer to avoid that if possible.

[verymilan]

Thanks a ton, great how quick that was! We will wait for the release including it :)

[reivilibre]

If you're not unhappy with running release candidates, the fix is in v1.78.0rc1 which has gone out today.

[verymilan]

Great, thanks for the info :)

[DMRobertson]

@verymilan were you able to purge this room when running the 1.78 release?