Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Goto binary serialization #2205

Conversation

remi-delmas-3000
Copy link
Contributor

@remi-delmas-3000 remi-delmas-3000 commented Feb 9, 2023

The last commit is from #2221 which is needed for regression runs to pass (will be cleaned up on rebase).

Description of changes:

This adds a new output mode --enable-unstable --write-goto-binary that produces the symbol table a goto binary instead of a JSON file, and skips the invocation of symtab2gb.

The new serializer uses value numbering to share identical strings and nodes in the binary file.

This feature depends on this CBMC PR diffblue/cbmc#7534, that makes it possible to load a goto binary produced by a different tool that may number strings differently from how CBMC numbers them.

Resolved issues:

Symtab2gb causes some noticeable overhead.

Call-outs:

The build scripts still need to be updated so that the feature is tested as part of regression runs (help needed).

Testing:

  • How is this change tested?
  • the module has internal tests and I ran the regression suite successfully with the option activated
  • Is this a refactor change?

No

Checklist

  • Each commit message has a non-empty body, explaining why the change was made
  • Methods or procedures are documented
  • Regression or unit tests are included, or existing tests cover the modified code
  • My PR is restricted to a single feature or bugfix

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.

Copy link
Contributor

@celinval celinval left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm so excited about this change! Thank you! Have you measured the performance improvement yet?

I haven't had time to review the serialization code yet.

I'm assuming you are planning to add a test as soon as the CBMC PR gets merged and released. If you don't mind, I'll block the merge till then.

@remi-delmas-3000
Copy link
Contributor Author

Thank you! Have you measured the performance improvement yet?

Serializing to GOTO binary seems to be as at least as fast or faster than when serializing to JSON, and we completely skip symtab2gb.

I'm assuming you are planning to add a test as soon as the CBMC PR gets merged and released.

The module has its own internal tests that run at every build.

As long as this is not used as default, would it be a good idea to have CI run the full regression suite a second time with GOTO binary output enabled (as done for the "unsound_experiments" feature) ?

I could add a couple of tests that specifically activate the switch but this would not be much compared to running the full regression suite.

@adpaco-aws
Copy link
Contributor

Serializing to GOTO binary seems to be as at least as fast or faster than when serializing to JSON, and we completely skip symtab2gb.

We would like to know about memory consumption as well. In fact, the biggest problem with symtab2gb (at least for Kani) is that launching several instances of it results in out-of-memory errors quite easily.

@remi-delmas-3000
Copy link
Contributor Author

remi-delmas-3000 commented Feb 10, 2023

some performance metrics measured when running scripts/std-lib-regression.sh

  • total runtime goes from 197s to 70s
  • peak memory consumption is reduced by two on the two largest crates proc_macro and std (see stats below)

crate std :

baseline

Command being timed: "/home/ubuntu/kani/target/kani/bin/kani-compiler --crate-name std --edition=2021 /home/ubuntu/.rustup/toolchains/nightly-2022-12-11-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type rlib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 --cfg feature="addr2line" --cfg feature="backtrace" --cfg feature="gimli-symbolize" --cfg feature="miniz_oxide" --cfg feature="object" --cfg feature="panic_unwind" --cfg feature="std_detect_dlsym_getauxval" --cfg feature="std_detect_file_io" -C metadata=4521dfe8186f7bad -C extra-filename=-4521dfe8186f7bad --out-dir /tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps --target x86_64-unknown-linux-gnu -Z force-unstable-if-unmarked -L dependency=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps -L dependency=/tmp/std_lib_test/target/debug/deps --extern addr2line=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libaddr2line-f4ba037b743064e3.rmeta --extern alloc=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/liballoc-d3e18ac105948c37.rmeta --extern cfg_if=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libcfg_if-fb37f524d97ee138.rmeta --extern compiler_builtins=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libcompiler_builtins-a5b7514ebb7379c4.rmeta --extern core=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libcore-672c5d05ab648bdb.rmeta --extern hashbrown=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libhashbrown-46445c7db4a1fb41.rmeta --extern libc=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/liblibc-7b17e0a66e9801f8.rmeta --extern miniz_oxide=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libminiz_oxide-b8e17be0cc86fe35.rmeta --extern object=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libobject-5c1de5f0d58e0ca9.rmeta --extern panic_abort=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libpanic_abort-82567b8412105d74.rmeta --extern panic_unwind=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libpanic_unwind-5ff4b67a1f33fae4.rmeta --extern rustc_demangle=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/librustc_demangle-15a62e4e79600337.rmeta --extern std_detect=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libstd_detect-2b7705bfffdedad6.rmeta --extern unwind=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libunwind-b20de0834f4fd475.rmeta --cap-lints allow --kani-compiler -Cpanic=abort -Cllvm-args=--goto-c -Cllvm-args=--ignore-global-asm -Cllvm-args=--reachability=legacy --cfg backtrace_in_libstd"
	User time (seconds): 78.80
	System time (seconds): 12.61
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:31.43
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 6588252
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 2293200
	Voluntary context switches: 7
	Involuntary context switches: 286
	Swaps: 0
	File system inputs: 0
	File system outputs: 1052376
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
    Finished dev [unoptimized + debuginfo] target(s) in 1m 50s

goto binary output

write_goto_binary_file statistics GotoBinarySerializerStats { 
  allocated_bytes: 308575992, 
  string_stats: SharingStats { 
    nof_unique: 172792, 
    min_count: 1, 
    min_id: Some(3), 
    max_count: 450153, 
    max_id: Some(183), 
    avg_count: 22.182531598684793 
  }, 
  irep_stats: SharingStats { 
    nof_unique: 1068436,
    min_count: 1,
    min_id: Some(4),
    max_count: 144211,
    max_id: Some(14),
    avg_count: 3.2993693585764947 
  } 
}
Command being timed: "/home/ubuntu/kani/target/kani/bin/kani-compiler --crate-name std --edition=2021 /home/ubuntu/.rustup/toolchains/nightly-2022-12-11-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type rlib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 --cfg feature="addr2line" --cfg feature="backtrace" --cfg feature="gimli-symbolize" --cfg feature="miniz_oxide" --cfg feature="object" --cfg feature="panic_unwind" --cfg feature="std_detect_dlsym_getauxval" --cfg feature="std_detect_file_io" -C metadata=4521dfe8186f7bad -C extra-filename=-4521dfe8186f7bad --out-dir /tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps --target x86_64-unknown-linux-gnu -Z force-unstable-if-unmarked -L dependency=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps -L dependency=/tmp/std_lib_test/target/debug/deps --extern addr2line=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libaddr2line-f4ba037b743064e3.rmeta --extern alloc=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/liballoc-d3e18ac105948c37.rmeta --extern cfg_if=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libcfg_if-fb37f524d97ee138.rmeta --extern compiler_builtins=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libcompiler_builtins-a5b7514ebb7379c4.rmeta --extern core=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libcore-672c5d05ab648bdb.rmeta --extern hashbrown=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libhashbrown-46445c7db4a1fb41.rmeta --extern libc=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/liblibc-7b17e0a66e9801f8.rmeta --extern miniz_oxide=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libminiz_oxide-b8e17be0cc86fe35.rmeta --extern object=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libobject-5c1de5f0d58e0ca9.rmeta --extern panic_abort=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libpanic_abort-82567b8412105d74.rmeta --extern panic_unwind=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libpanic_unwind-5ff4b67a1f33fae4.rmeta --extern rustc_demangle=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/librustc_demangle-15a62e4e79600337.rmeta --extern std_detect=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libstd_detect-2b7705bfffdedad6.rmeta --extern unwind=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libunwind-b20de0834f4fd475.rmeta --cap-lints allow --kani-compiler -Cpanic=abort -Cllvm-args=--goto-c -Cllvm-args=--ignore-global-asm -Cllvm-args=--reachability=legacy --cfg backtrace_in_libstd"
	User time (seconds): 15.83
	System time (seconds): 9.66
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:25.50
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 2759024
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 1006544
	Voluntary context switches: 3
	Involuntary context switches: 75
	Swaps: 0
	File system inputs: 0
	File system outputs: 118224
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
    Finished dev [unoptimized + debuginfo] target(s) in 44.50s

crate proc_macro

baseline

Command being timed: "/home/ubuntu/kani/target/kani/bin/kani-compiler --crate-name proc_macro --edition=2021 /home/ubuntu/.rustup/toolchains/nightly-2022-12-11-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/proc_macro/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 -C metadata=e3dad344b94a38f1 -C extra-filename=-e3dad344b94a38f1 --out-dir /tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps --target x86_64-unknown-linux-gnu -Z force-unstable-if-unmarked -L dependency=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps -L dependency=/tmp/std_lib_test/target/debug/deps --extern core=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libcore-672c5d05ab648bdb.rmeta --extern std=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libstd-4521dfe8186f7bad.rmeta --cap-lints allow --kani-compiler -Cpanic=abort -Cllvm-args=--goto-c -Cllvm-args=--ignore-global-asm -Cllvm-args=--reachability=legacy"
	User time (seconds): 17.49
	System time (seconds): 3.58
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:21.08
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 1737428
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 610241
	Voluntary context switches: 6
	Involuntary context switches: 66
	Swaps: 0
	File system inputs: 0
	File system outputs: 284512
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

with binary output

write_goto_binary_file statistics GotoBinarySerializerStats { 
  allocated_bytes: 76982280, 
  string_stats: SharingStats { 
    nof_unique: 49944, 
    min_count: 1, 
    min_id: Some(3), 
    max_count: 109757, 
    max_id: Some(177), 
    avg_count: 19.066154092583663 
  }, 
  irep_stats: SharingStats { 
    nof_unique: 269647,
    min_count: 1,
    min_id: Some(4),
    max_count: 39254,
    max_id: Some(14),
    avg_count: 3.2148327257488507 
  } 
}
Command being timed: "/home/ubuntu/kani/target/kani/bin/kani-compiler --crate-name proc_macro --edition=2021 /home/ubuntu/.rustup/toolchains/nightly-2022-12-11-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/proc_macro/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 -C metadata=e3dad344b94a38f1 -C extra-filename=-e3dad344b94a38f1 --out-dir /tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps --target x86_64-unknown-linux-gnu -Z force-unstable-if-unmarked -L dependency=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps -L dependency=/tmp/std_lib_test/target/debug/deps --extern core=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libcore-672c5d05ab648bdb.rmeta --extern std=/tmp/std_lib_test/target/x86_64-unknown-linux-gnu/debug/deps/libstd-4521dfe8186f7bad.rmeta --cap-lints allow --kani-compiler -Cpanic=abort -Cllvm-args=--goto-c -Cllvm-args=--ignore-global-asm -Cllvm-args=--reachability=legacy"
	User time (seconds): 3.90
	System time (seconds): 2.77
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:06.68
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	 Maximum resident set size (kbytes): 811320
    Maximum resident set size (kbytes): 1737428	
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 280214
	Voluntary context switches: 2
	Involuntary context switches: 24
	Swaps: 0
	File system inputs: 0
	File system outputs: 28328
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

Copy link
Contributor

@zhassan-aws zhassan-aws left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work @remi-delmas-3000! Looks very promising! I have a few comments.

kani-compiler/kani_queries/src/lib.rs Outdated Show resolved Hide resolved
cprover_bindings/src/irep/goto_binary_serde.rs Outdated Show resolved Hide resolved
cprover_bindings/src/irep/goto_binary_serde.rs Outdated Show resolved Hide resolved
cprover_bindings/src/irep/goto_binary_serde.rs Outdated Show resolved Hide resolved
cprover_bindings/src/irep/goto_binary_serde.rs Outdated Show resolved Hide resolved
cprover_bindings/src/irep/goto_binary_serde.rs Outdated Show resolved Hide resolved
@remi-delmas-3000 remi-delmas-3000 force-pushed the goto-binary-serialization branch from 0d8adb0 to 3bb7e3a Compare February 14, 2023 16:09
@remi-delmas-3000
Copy link
Contributor Author

@celinval @zhassan-aws @adpaco-aws Update: the PR diffblue/cbmc#7534 that allows CBMC to load the binaries produced by Kani has been merged, can we consider merging this one now too ?

@celinval
Copy link
Contributor

@celinval @zhassan-aws @adpaco-aws Update: the PR diffblue/cbmc#7534 that allows CBMC to load the binaries produced by Kani has been merged, can we consider merging this one now too ?

That's great! Can you please add some tests to the PR? One suggestion is to create a new CI job that runs all tests using the new mode.

What would be the next steps for enabling this logic by default?

@zhassan-aws
Copy link
Contributor

zhassan-aws commented Feb 15, 2023

can we consider merging this one now too ?

Do we need to wait for a new CBMC release?

Yes I think we do.

@remi-delmas-3000 remi-delmas-3000 force-pushed the goto-binary-serialization branch 4 times, most recently from 3f193d1 to 5fa1074 Compare February 20, 2023 19:47
@remi-delmas-3000
Copy link
Contributor Author

This is now final, but depends on #2221 for the regression run to pass.

Copy link
Contributor

@zhassan-aws zhassan-aws left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still reviewing, but a couple of initial comments.

.github/workflows/kani.yml Outdated Show resolved Hide resolved
@remi-delmas-3000 remi-delmas-3000 force-pushed the goto-binary-serialization branch from 5fa1074 to bad2ab7 Compare February 21, 2023 15:19
@remi-delmas-3000
Copy link
Contributor Author

I'm so excited about this change! Thank you! Have you measured the performance improvement yet?

I haven't had time to review the serialization code yet.

I'm assuming you are planning to add a test as soon as the CBMC PR gets merged and released. If you don't mind, I'll block the merge till then.

@celinval can we unblock it now that regression tests are in place?

@celinval celinval dismissed their stale review February 21, 2023 21:31

Thanks for adding tests

@remi-delmas-3000 remi-delmas-3000 force-pushed the goto-binary-serialization branch from ca968b8 to 893cf3b Compare February 21, 2023 22:17
Copy link
Contributor

@zhassan-aws zhassan-aws left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @remi-delmas-3000! Mostly minor comments.

cprover_bindings/src/irep/goto_binary_serde.rs Outdated Show resolved Hide resolved
cprover_bindings/src/irep/goto_binary_serde.rs Outdated Show resolved Hide resolved
cprover_bindings/src/irep/goto_binary_serde.rs Outdated Show resolved Hide resolved
cprover_bindings/src/irep/goto_binary_serde.rs Outdated Show resolved Hide resolved
cprover_bindings/src/irep/goto_binary_serde.rs Outdated Show resolved Hide resolved
cprover_bindings/src/irep/goto_binary_serde.rs Outdated Show resolved Hide resolved
cprover_bindings/src/irep/goto_binary_serde.rs Outdated Show resolved Hide resolved
cprover_bindings/src/irep/goto_binary_serde.rs Outdated Show resolved Hide resolved
@remi-delmas-3000 remi-delmas-3000 force-pushed the goto-binary-serialization branch 2 times, most recently from 489a88f to 636c708 Compare March 2, 2023 18:06
kani-compiler/kani_queries/src/lib.rs Outdated Show resolved Hide resolved
kani-driver/src/call_single_file.rs Outdated Show resolved Hide resolved
@remi-delmas-3000 remi-delmas-3000 force-pushed the goto-binary-serialization branch from 636c708 to ca1c323 Compare March 2, 2023 20:42
@remi-delmas-3000
Copy link
Contributor Author

The latest commit makes goto binary export the default, adds a switch --write-json-symtab to force JSON symtab export, and adds a compile-time feature "write_json_symtab" that forces JSON export as the default at compile time. The feature is activated in the kani.yml file for regression runs

Remi Delmas added 13 commits March 2, 2023 20:45
Update Cargo.lock with memuse
Modified `call_single_file.rs` so that write_goto_binary
can be enabled by defining the KANI_ENABLE_WRITE_GOTO_BINARY
env variable.
When `--write-goto-binary` is used, JSON symtab files are not produced, so we should not rely on them.
For `--gen-c` we now store the `name` to `prettyName` mapping in its own JSON `ArtifactType::PrettyNameMap`.
For `--function`, we modify `CargoOutputs` so that it tracks GOTO symtabs instead of JSON symtabs.
- add CLI flag to reenable JSON symtab
- add compile-time feature to force JSON symtabs
@remi-delmas-3000 remi-delmas-3000 force-pushed the goto-binary-serialization branch from ca1c323 to 57c38bc Compare March 2, 2023 20:45
kani-compiler/src/kani_compiler.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@celinval celinval left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few notes... Thanks for doing this

cprover_bindings/Cargo.toml Outdated Show resolved Hide resolved
cprover_bindings/src/irep/goto_binary_serde.rs Outdated Show resolved Hide resolved
cprover_bindings/src/irep/goto_binary_serde.rs Outdated Show resolved Hide resolved
@remi-delmas-3000 remi-delmas-3000 force-pushed the goto-binary-serialization branch from 2e07744 to ffd7f0d Compare March 7, 2023 19:23
Copy link
Contributor

@celinval celinval left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome. It's such a great improvement. Thanks for doing this. As I mentioned offline, I'm happy to merge this as is and we can prune it as we go.

@remi-delmas-3000 remi-delmas-3000 merged commit fca5e92 into model-checking:main Mar 7, 2023
@remi-delmas-3000 remi-delmas-3000 deleted the goto-binary-serialization branch March 7, 2023 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[C] Feature / Enhancement A new feature request or enhancement to an existing feature. [E] Performance Track performance improvement (Time / Memory / CPU)
Projects
No open projects
Status: In Progress
4 participants