Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable oppportunistic fd counting fast path #234

Merged
merged 2 commits into from
Jan 15, 2023

Conversation

bobrik
Copy link
Contributor

@bobrik bobrik commented Jan 14, 2023

Existing slow path takes ~52000us of CPU time on my laptop to count 100k open files. Compare this to just 13us for the baseline, when no extra files are open by a program:

fd count = 7, elapsed = 13 us
fd count = 100007, elapsed = 51863 us

Adding fastpath from Linux v6.2 makes it fast:

fd count = 4, elapsed = 2 us
fd count = 100004, elapsed = 8 us

This is before taking in account any lock contention effects in the kernel if you try to count files from multiple threads concurrently, which makes the slow path even slower, burning a lot more CPU in the process. See:

I used the following code to benchmark:

use procfs::process::Process;
use std::{fs::File, time::Instant};

fn main() {
    let myself = Process::myself().unwrap();

    let started = Instant::now();
    let count = myself.fd_count().unwrap();
    let elapsed_us = started.elapsed().as_micros();

    eprintln!("fd count = {}, elapsed = {} us", count, elapsed_us);

    let files = (0..100_000)
        .map(|_| File::open("/etc/hosts").unwrap())
        .collect::<Vec<_>>();

    let started = Instant::now();
    let count = myself.fd_count().unwrap();
    let elapsed_us = started.elapsed().as_micros();

    eprintln!("fd count = {}, elapsed = {} us", count, elapsed_us);

    drop(files);
}

See also: prometheus/procfs#486

Existing slow path takes ~52000us of CPU time on my laptop to count
100k open files. Compare this to just 13us for the baseline,
when no extra files are open by a program:

```
fd count = 7, elapsed = 13 us
fd count = 100007, elapsed = 51863 us
```

Adding fastpath from Linux v6.2 makes it fast:

```
fd count = 4, elapsed = 2 us
fd count = 100004, elapsed = 8 us
```

This is before taking in account any lock contention effects in the kernel
if you try to count files from multiple threads concurrently, which makes
the slow path even slower, burning a lot more CPU in the process. See:

* torvalds/linux@f1f1f2569901

I used the following code to benchmark:

```rust
use procfs::process::Process;
use std::{fs::File, time::Instant};

fn main() {
    let myself = Process::myself().unwrap();

    let started = Instant::now();
    let count = myself.fd_count().unwrap();
    let elapsed_us = started.elapsed().as_micros();

    eprintln!("fd count = {}, elapsed = {} us", count, elapsed_us);

    let files = (0..100_000)
        .map(|_| File::open("/etc/hosts").unwrap())
        .collect::<Vec<_>>();

    let started = Instant::now();
    let count = myself.fd_count().unwrap();
    let elapsed_us = started.elapsed().as_micros();

    eprintln!("fd count = {}, elapsed = {} us", count, elapsed_us);

    drop(files);
}
```
@eminence
Copy link
Owner

Looks nice, thanks. I just realized that we don't have any tests for fd_count()... would you mind adding a small commit that calls fd_count -- probably best in test_all() or test_proc_fds()

@bobrik
Copy link
Contributor Author

bobrik commented Jan 15, 2023

Sure, I added a basic test to make sure that the count increments and decrements as expected. It needs to run single threaded to avoid picking up fds from other threads. Let me know if you had something else in mind.

@eminence
Copy link
Owner

That's perfect, thank you!

@eminence eminence merged commit e441e71 into eminence:master Jan 15, 2023
@bobrik bobrik deleted the ivan/fastpath branch March 23, 2023 18:19
@bobrik bobrik restored the ivan/fastpath branch March 23, 2023 18:19
@bobrik bobrik deleted the ivan/fastpath branch March 23, 2023 18:20
@bobrik bobrik restored the ivan/fastpath branch March 23, 2023 18:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants