OverflowError in ParticleBitmap

[weiguangcui]

Bug report

Bug summary

  dd      = ds.all_data()
  File "/home/weiguang/.local/lib/python3.8/site-packages/yt/data_objects/static_output.py", line 1199, in all_data
    self.index
  File "/home/weiguang/.local/lib/python3.8/site-packages/yt/data_objects/static_output.py", line 609, in index
    self._instantiated_index = self._index_class(
  File "/home/weiguang/.local/lib/python3.8/site-packages/yt/geometry/particle_geometry_handler.py", line 73, in __init__
    self._initialize_index()
  File "/home/weiguang/.local/lib/python3.8/site-packages/yt/frontends/sph/data_structures.py", line 89, in _initialize_index
    super()._initialize_index()
  File "/home/weiguang/.local/lib/python3.8/site-packages/yt/geometry/particle_geometry_handler.py", line 213, in _initialize_index
    self.regions = ParticleBitmap(
  File "yt/geometry/particle_oct_container.pyx", line 487, in yt.geometry.particle_oct_container.ParticleBitmap.__init__
OverflowError: int too big to convert

Code for reproduction

This is a large simulation, with 512 snapshot files and about 5*10^10 particles in the whole simulation.

ds=yt.load('simulation_snapshot')
dd      = ds.all_data()

Version Information

• Operating System: Linux
• Python Version: 3.8.5
• yt version: 4.2.0 (latest, installed with pip)

Thank you for your help.

Best,
Weiguang

edit by @neutrinoceros : added markup to the python-traceback block

[neutrinoceros]

Hi @weiguangcui , thank you for reporting this

First of all, I see that the line where the crash occurs was added in #4307, so it may be a regression. Can you please try downgrading yt to 4.1.4 and see if it works better ?

That said, I'm surprised this crashes, given that this line is attempting to convert an np.int64_t to bytes, and this int represents the number of files, so it should be only 512, well within the int64 range.
Furthermore, I don't understand where the invoked method to_bytes is defined from, or where to find documentation for it. @Xarthisius, can you help with that ?

[weiguangcui]

Hi @neutrinoceros, I can confirm that yt@4.1.4 works totally fine, though it is not fully done (taken too much time to load all these particles). The error did not show up as immediately in the previous case.

[neutrinoceros]

I think the to_bytes method is defined on Python integers here: https://docs.python.org/3/library/stdtypes.html?highlight=to_bytes#int.to_bytes

It's expected that 512 overflows a 2 bytes representation. (edit: I was confusing "byte" and "bit" here, they sound identical in my mother language ! 2 bytes is however not sufficient to represent an int64 in the general case, so I think my conclusions still stand ?)

I assume the correct fix would simply be to allocate more bytes to this part of the hash: 64 bits (8 bytes) would be needed to completely remove the risk of seeing that error pop up again. Other parts of the hash could also use similar patches (I see index_order1 and index_order2 are int32_t, but are currently converted to only 2-bytes representation).
I'll attempt a patch.

[neutrinoceros]

@weiguangcui I've opened #4472 to fix this. Can you try building yt from this branch and see if it fixes your issue ?

This command should suffice to build with my patch:

python -m pip install git+/~https://github.com/neutrinoceros/yt.git@fix_particle_bitmap_hash

[neutrinoceros]

my patch has been accepted and will be part of yt 4.2.1, in the mean time you can install yt from source with

python -m pip install git+/~https://github.com/yt-project/yt.git

and of course, you are still welcome to report any problems ! thank you again !

[weiguangcui]

Sorry for the late reply. But I can confirm that the latest version has no problem with that any more.
Please close this ticket as you wish.