Cannot Search Chinese Words Correctly in E2EE Rooms

[uetcis]

Description

Cannot search Chinese words correctly in E2E rooms.

Steps to reproduce

• Join an E2EE room and send some Chinese words
• Try searching
• Receive incomplete results (usually no results at all)

Describe how what happens differs from what you expected.

Client local searching in E2EE rooms should work for non-Latin languages while it doesn't.

Version information

• Platform: Both

For the web app:

• Browser: Safari Version 14.0.2 (16610.3.7.1.9)
• OS: macOS
• URL: app.element.io (1.7.18)

For the desktop app:

• OS: macOS
• Version: Version 1.7.18 (1.7.18)

[jryans]

Thanks for the report. This is tied to tokenisation issues in our indexer (matrix-org/seshat#7).