Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sort operation used more than the maximum 33554432 bytes of RAM #8

Open
bodqhrohro opened this issue Jan 17, 2021 · 0 comments
Open

Comments

@bodqhrohro
Copy link
Contributor

I encountered this when exporting a large chat (137988 messages):

Traceback (most recent call last):
  File "/media/d/temp/git/matrix-archive/export_messages.py", line 80, in <module>
    export_archive()
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/media/d/temp/git/matrix-archive/export_messages.py", line 66, in export_archive
    print(f"Writing {len(messages)} messages to {filename!r}")
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/mongoengine/queryset/queryset.py", line 62, in __len__
    list(self._iter_results())
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/mongoengine/queryset/queryset.py", line 110, in _iter_results
    self._populate_cache()
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/mongoengine/queryset/queryset.py", line 129, in _populate_cache
    self._result_cache.append(next(self))
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/mongoengine/queryset/base.py", line 1619, in __next__
    raw_doc = next(self._cursor)
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/pymongo/cursor.py", line 1207, in next
    if len(self.__data) or self._refresh():
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/pymongo/cursor.py", line 1124, in _refresh
    self.__send_message(q)
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/pymongo/cursor.py", line 999, in __send_message
    response = client._run_operation_with_response(
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1368, in _run_operation_with_response
    return self._retryable_read(
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1471, in _retryable_read
    return func(session, server, sock_info, slave_ok)
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1360, in _cmd
    return server.run_operation_with_response(
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/pymongo/server.py", line 135, in run_operation_with_response
    _check_command_response(first, sock_info.max_wire_version)
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/pymongo/helpers.py", line 160, in _check_command_response
    raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: Executor error during find command :: caused by :: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit., full error: {'ok': 0.0, 'errmsg': 'Executor error during find command :: caused by :: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.', 'code': 96, 'codeName': 'OperationFailed'}

This can be worked around, though adding an actual index should be better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant