Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

don't allocate string when writing to csv #935

Merged
merged 2 commits into from
Apr 10, 2022

Conversation

ritchie46
Copy link
Collaborator

@ritchie46 ritchie46 commented Apr 7, 2022

Writing to csv is pretty slow. Polars can read two datetime columns in 500 ms, whereas writing the same data to csv takes 33.6 seconds.

Upon looking at the code I saw we allocate a String for every value we format.

This PR makes sure we write immediately to the Vec<u8> buffer.

@codecov
Copy link

codecov bot commented Apr 7, 2022

Codecov Report

Merging #935 (62d8263) into main (36b0824) will increase coverage by 0.01%.
The diff coverage is 88.88%.

@@            Coverage Diff             @@
##             main     #935      +/-   ##
==========================================
+ Coverage   71.10%   71.12%   +0.01%     
==========================================
  Files         346      346              
  Lines       18951    18950       -1     
==========================================
+ Hits        13476    13479       +3     
+ Misses       5475     5471       -4     
Impacted Files Coverage Δ
src/io/csv/write/serialize.rs 61.04% <88.88%> (+1.44%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 36b0824...62d8263. Read the comment docs.

@ritchie46 ritchie46 force-pushed the improve_csv_write branch from ae72f52 to 74b46f5 Compare April 7, 2022 15:02
@ritchie46 ritchie46 force-pushed the improve_csv_write branch from 74b46f5 to 62d8263 Compare April 7, 2022 15:42
@ritchie46
Copy link
Collaborator Author

The clippy lints seem unrelated to these changes.

@jorgecarleitao jorgecarleitao merged commit 4df7032 into jorgecarleitao:main Apr 10, 2022
@jorgecarleitao jorgecarleitao added the enhancement An improvement to an existing feature label Apr 10, 2022
@jorgecarleitao
Copy link
Owner

oof, thanks a lot, @ritchie46 , really good stuff!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement An improvement to an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants