Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

polars 1.17 can not save file to RamDisk #20231

Closed
2 tasks done
wukan1986 opened this issue Dec 9, 2024 · 5 comments · Fixed by #20245
Closed
2 tasks done

polars 1.17 can not save file to RamDisk #20231

wukan1986 opened this issue Dec 9, 2024 · 5 comments · Fixed by #20245
Assignees
Labels
accepted Ready for implementation bug Something isn't working P-low Priority: low python Related to Python Polars regression Issue introduced by a new release

Comments

@wukan1986
Copy link

wukan1986 commented Dec 9, 2024

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl

df = pl.DataFrame({'A': [1, 2, 3, 4, 5], 'B': ['a', 'b', 'c', 'd', 'e']})

df.write_parquet(r'd:\test.parquet')  # SSD

df.write_parquet(r'M:\test.parquet')  # ImDisk Toolkit
r"""
Traceback (most recent call last):
  File "D:\GitHub\ddump\examples\jqresearch\t1.py", line 6, in <module>
    df.write_parquet(r'M:\test.parquet')  # ImDisk Toolkit
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\Kan\miniconda3\envs\py311_1\Lib\site-packages\polars\dataframe\frame.py", line 4038, in write_parquet
    self._df.write_parquet(
OSError: 函数不正确。 (os error 1)
"""

df.write_csv(r'N:\test.csv')  # Rubik's Cube RamDisk
r"""
Traceback (most recent call last):
  File "D:\GitHub\ddump\examples\jqresearch\t1.py", line 16, in <module>
    df.write_csv(r'N:\test.csv')  # Rubik's Cube RamDisk
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\Kan\miniconda3\envs\py311_1\Lib\site-packages\polars\dataframe\frame.py", line 2983, in write_csv
    self._df.write_csv(
FileNotFoundError: 系统找不到指定的文件。 (os error 2)
"""

I used to save files on RamDisk, but after updating from 1.16 to 1.17, I found that the files could not be saved anymore.
I tested two RamDisk software, but neither of them works

ImDisk Toolkit
https://imdisk-toolkit.en.lo4d.com/download

Log output

No response

Issue description

polars 1.17 can not save file to RamDisk

Expected behavior

save file to RamDisk

Installed versions

--------Version info---------
Polars:              1.17.1
Index type:          UInt32
Platform:            Windows-10-10.0.22631-SP0
Python:              3.11.3 | packaged by Anaconda, Inc. | (main, Apr 19 2023, 23:46:34) [MSC v.1916 64 bit (AMD64)]
LTS CPU:             False

----Optional dependencies----
adbc_driver_manager  <not installed>
altair               5.0.1
boto3                <not installed>
cloudpickle          <not installed>
connectorx           <not installed>
deltalake            <not installed>
fastexcel            <not installed>
fsspec               <not installed>
gevent               <not installed>
google.auth          2.26.1
great_tables         <not installed>
matplotlib           3.7.1
nest_asyncio         1.5.6
numpy                1.26.4
openpyxl             3.1.2
pandas               2.2.3
pyarrow              12.0.0
pydantic             1.10.7
pyiceberg            <not installed>
sqlalchemy           2.0.13
torch                2.0.1+cu118
xlsx2csv             <not installed>
xlsxwriter           3.2.0
@wukan1986 wukan1986 added bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars labels Dec 9, 2024
@nameexhaustion nameexhaustion added regression Issue introduced by a new release P-low Priority: low labels Dec 10, 2024
@nameexhaustion nameexhaustion removed the needs triage Awaiting prioritization by a maintainer label Dec 10, 2024
@github-project-automation github-project-automation bot moved this to Ready in Backlog Dec 10, 2024
@nameexhaustion
Copy link
Collaborator

nameexhaustion commented Dec 10, 2024

I cannot reproduce this - I've tested on a mount point other than the C drive. Can you show the logs when you run after setting POLARS_VERBOSE=1 in the environment variables?

@wukan1986
Copy link
Author

wukan1986 commented Dec 10, 2024

can not show log

import polars as pl

print(os.environ.get('POLARS_VERBOSE'))  # 1

df = pl.DataFrame({'A': [1, 2, 3, 4, 5], 'B': ['a', 'b', 'c', 'd', 'e']})

# df.write_parquet(r'd:\test.parquet')  # SSD
df.write_parquet(r'M:\test.parquet')  # ImDisk Toolkit
(py311_1) D:\GitHub\ddump\tests>python t1.py
1
Traceback (most recent call last):
  File "D:\GitHub\ddump\tests\t1.py", line 10, in <module>
    df.write_parquet(r'M:\test.parquet')  # ImDisk Toolkit
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\Kan\miniconda3\envs\py311_1\Lib\site-packages\polars\dataframe\frame.py", line 4038, in write_parquet
    self._df.write_parquet(
OSError: 函数不正确。 (os error 1)

It create a empty file, then throw error

M:\>dir
 驱动器 M 中的卷是 RamDisk
 卷的序列号是 E62C-C447

 M:\ 的目录

2024/12/10  16:45                 0 test.parquet
               1 个文件              0 字节
               0 个目录  8,555,302,912 可用字节

@wukan1986
Copy link
Author

I have tried on two computers and they both reported errors

Could it be caused by this place?

#20101

@wukan1986
Copy link
Author

Hi, @nameexhaustion

use std::path::{Path};


fn main() {
    let path = Path::new("D:\\test.csv");

    eprintln!(
        "try_get_writeable: local: {} (canonicalize: {:?})",
        path.to_str().unwrap(),
        std::fs::canonicalize(&path)
    );

    let path = Path::new("M:\\test.csv");

    eprintln!(
        "try_get_writeable: local: {} (canonicalize: {:?})",
        path.to_str().unwrap(),
        std::fs::canonicalize(&path)
    );
}
warning: `C:\Users\Kan\.cargo\config` is deprecated in favor of `config.toml`
note: if you need to support cargo 1.38 or earlier, you can symlink `config` to `config.toml`
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.01s
     Running `target\debug\rust-test.exe`
try_get_writeable: local: D:\test.csv (canonicalize: Ok("\\\\?\\D:\\test.csv"))
try_get_writeable: local: M:\test.csv (canonicalize: Err(Os { code: 1, kind: Uncategorized, message: "函数不正确。" }))

M drive is create by ImDisk Toolkit

@nameexhaustion
Copy link
Collaborator

I see, in that case it should work after the PR is merged

@github-project-automation github-project-automation bot moved this from Ready to Done in Backlog Dec 10, 2024
@c-peters c-peters added the accepted Ready for implementation label Dec 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
accepted Ready for implementation bug Something isn't working P-low Priority: low python Related to Python Polars regression Issue introduced by a new release
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants