qsv 0.134.0

[BrewTestBot]

Created by brew bump

---

Created with brew bump-formula-pr.

release notes

  
  
  


  
  
  


qsv pro v1 is here! 🎉
If you've been using qsv for a while, even if you're a command-line ninja, you'll find a lot of new capabilities in qsv pro that can make your data wrangling experience even better!
Apart from making qsv easier to use, qsv pro has a multitude of features including: view interactive data tables; browse stats/frequency/metadata; run recipes and tools (scripts); run Polars SQL queries; an interface using Retrieval Augmented Generation (RAG) techniques to attempt converting Natural Language queries to Polars SQL queries; regular expression search; export to multiple file formats; download/upload from/to compatible CKAN instances; design custom node-based flows and data pipelines; interact with a local API from external programs including the qsv pro command; run various qsv commands in a graphical user interface; and the list goes on!
That's just the beginning, there's more to come! You just have to try it!
Download qsv pro v1 now at qsvpro.dathere.com.
Other highlights include:

pro: new command to allow qsv to interact with the qsv pro API to tap into qsv pro exclusive features.
lens: new command to interactively view CSVs using the csvlens crate.
The ludicrously fast diff command is now easier to use with its --drop-equal-fields option. @janriemer continues to work on his csv-diff crate, and there's more diff UX improvements coming soon!
stats adds sum_length and avg_length "streaming" statistics in addition to the existing min_length and max_length metrics. These are especially useful for datasets with a lot of "free text" columns.
stats also got "smarter" and "faster" by dog-fooding its own statistics to make it run faster!

It's a little complicated, but the way stats works is that it compiles the "streaming" statistics on the fly first, and the more expensive advanced statistics are "lazily" computed at the end.

Since we now compile "sort order" in a streaming manner, we use this info when deriving cardinality at the end to see if we can skip sorting - an otherwise necessary step to get cardinality which is done by "scanning" all the sorted values of a column. Everytime two neighboring values differ in a sorted column, it increments the cardinality count.

Apart from this "sort order" optimization, we also improved the "cardinality scan" algorithm - halving its memory footprint and making it faster still for larger datasets by parallelizing the computation!

This in turn, makes the frequency command faster and more memory efficient!
we now also use our own fork of the csv crate, featuring SIMD-accelerated UTF-8 validation and other minor perf tweaks, making the entire qsv suite faster still!


Added

pro: add qsv pro command to interact with qsv pro API by @rzmk in pro: add qsv pro command to interact with qsv pro API dathere/qsv#2039
lens: new command to interactively view CSVs using the csvlens crate lens: new command to interactively view CSVs using the csvlens crate dathere/qsv#2117
apply: add crc32 operation apply: add crc32 operation dathere/qsv#2121
count: add --delimiter option count: add --delimiter option dathere/qsv#2120
diff: add flag --drop-equal-fields by @janriemer in diff: add flag --drop-equal-fields dathere/qsv#2114
stats: add sum_length and avg_length columns stats: add sum_length and avg_length columns dathere/qsv#2113
stats: smarter cardinality computation - added new parallel algorithm for large datasets (10,000+ rows) and updated sequential algorithm for smaller datasets dathere/qsv@4e63fec

Changed

count: added comment to justify magic number dathere/qsv@5241e39
stats: use simdjson for faster JSONL parsing; micro-optimize compute hot loop dathere/qsv@0e8b734
stats: standardized OVERFLOW and UNDERFLOW messages dathere/qsv@38c6128
sort: renamed symbol so eliminate devskim lint false positive warning dathere/qsv@12db739
enable lens feature in GH workflows enable lens feature in GH workflows dathere/qsv#2122
deps: bump polars 0.42.0 to latest upstream at time of release dathere/qsv@3c17ed1
deps: use our own optimized fork of csv crate, with simdutf8 validation and other minor perf tweaks dathere/qsv@e4bcd71
build(deps): bump serde from 1.0.209 to 1.0.210 by @dependabot in build(deps): bump serde from 1.0.209 to 1.0.210 dathere/qsv#2111
build(deps): bump serde_json from 1.0.127 to 1.0.128 by @dependabot in build(deps): bump serde_json from 1.0.127 to 1.0.128 dathere/qsv#2106
build(deps): bump qsv-stats from 0.19.0 to 0.22.0 build(deps): bump qsv-stats from 0.19.0 to 0.19.1 dathere/qsv#2107 build(deps): bump qsv-stats from 0.19.1 to 0.20.0 dathere/qsv#2112 dathere/qsv@cb1eb60
apply select clippy lint suggestions
updated several indirect dependencies
made various doc and usage text improvements

Fixed

schema: Print an error if the qsv stats invocation fails by @abrauchli in schema: Print an error if the qsv stats invocation fails dathere/qsv#2110

New Contributors

@abrauchli made their first contribution in schema: Print an error if the qsv stats invocation fails dathere/qsv#2110

Full Changelog: dathere/qsv@0.133.1...0.134.0

[github-actions]

🤖 An automated task has requested bottles to be published to this PR.