Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Fix index out of bounds in uniform_hist_count #20133

Merged
merged 3 commits into from
Dec 4, 2024

Conversation

iharthi
Copy link
Contributor

@iharthi iharthi commented Dec 3, 2024

Sometimes, index out of bounds happened in uniform_hist_count in cases when due to floating point representation imprecision idx value went over the maximum index (num_bins - 1). Example error message:

thread 'polars-4' panicked at crates/polars-ops/src/chunked_array/hist.rs:112:22:
index out of bounds: the len is 5 but the index is 5

To work around the issue, add an explicit check that forces value into expected number of intervals. I am not entirely sure this is mathematically correct solution, but it seems to work well.

I added a test that fails on current main branch on my machine just in case, but I know for sure it does not fail with error in other environments (like google colab, for example) .

@iharthi iharthi changed the title fix(rust): fix index out of bounds in uniform_hist_count fix(rust): Fix index out of bounds in uniform_hist_count Dec 3, 2024
@github-actions github-actions bot added fix Bug fix rust Related to Rust Polars and removed title needs formatting labels Dec 3, 2024
@iharthi iharthi force-pushed the fix-hist-index-out-of-bounds branch from 4f00e8e to 4a5036a Compare December 3, 2024 15:05
Copy link

codecov bot commented Dec 3, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.57%. Comparing base (ae05c68) to head (12e5b3a).
Report is 4 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #20133      +/-   ##
==========================================
+ Coverage   79.53%   79.57%   +0.03%     
==========================================
  Files        1562     1563       +1     
  Lines      217148   217396     +248     
  Branches     2465     2472       +7     
==========================================
+ Hits       172707   172985     +278     
+ Misses      43880    43843      -37     
- Partials      561      568       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ritchie46
Copy link
Member

Thanks, could you add a minimal test on the python side?

Sometimes, index out of bounds happened in uniform_hist_count
in cases when due to floating point representation imprecision
idx value went over the maximum index (num_bins - 1).
@iharthi iharthi force-pushed the fix-hist-index-out-of-bounds branch from 0aefdbf to 00688ad Compare December 3, 2024 16:26
@ritchie46 ritchie46 merged commit c346fc2 into pola-rs:main Dec 4, 2024
25 checks passed
@ritchie46 ritchie46 changed the title fix(rust): Fix index out of bounds in uniform_hist_count fix: Fix index out of bounds in uniform_hist_count Dec 4, 2024
@github-actions github-actions bot added the python Related to Python Polars label Dec 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix Bug fix python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants