Code

Parquet streaming [WIP]#538

Code

Open

knighton wants to merge 13 commits intodevmosaicml/streaming:devfrom james/streaming-parquetmosaicml/streaming:james/streaming-parquetCopy head branch name to clipboard

+5,139-3,176

Commits

Commits on Nov 11, 2023

• Merge branch 'main' of github.com:mosaicml/streaming

  knighton

  committed
  a51288f

  Copy full SHA for a51288f

Commits on Nov 18, 2023

• Move examples out, merge base/ upward (#494)

  Show description for 7f5d160
  knighton

  authored
  7f5d160

  Copy full SHA for 7f5d160

Commits on Dec 7, 2023

• Dataset kwargs switchover. (#523)

  Show description for 3cd8a22
  knighton

  authored
  3cd8a22

  Copy full SHA for 3cd8a22

Commits on Dec 12, 2023

• Organize utils. (#524)

  Show description for bf81f6b
  

  knighton

  and

  karan6181

  authored
  bf81f6b

  Copy full SHA for bf81f6b

• Fancy overly long lines command. (#529)

  knighton

  authored
  0ecf06f

  Copy full SHA for 0ecf06f

Commits on Dec 14, 2023

• Redo/generalize/tighten args shorthand (#530)

  Show description for 7c3fa05
  knighton

  authored
  7c3fa05

  Copy full SHA for 7c3fa05

Commits on Dec 15, 2023

• Add benchmarking suite for all backends and formats (#533)

  Show description for d969cd6
  knighton

  authored
  d969cd6

  Copy full SHA for d969cd6

• Replicate allow_unsafe_types in dev, where it will be used more. (#534)

  knighton

  authored
  78c150e

  Copy full SHA for 78c150e

• New storage APIs (#536)

  Show description for 02bd910
  knighton

  authored
  02bd910

  Copy full SHA for 02bd910

• Improve naming: JSON shards are actually JSONL, etc. (#537)

  Show description for 3972c9d
  knighton

  authored
  3972c9d

  Copy full SHA for 3972c9d

• Parquet ingestion, Streaming Parquet shards (draft).

  knighton

  committed
  3dcf500

  Copy full SHA for 3dcf500

• Fix (docstrings).

  knighton

  committed
  6d4bb55

  Copy full SHA for 6d4bb55

• Fix (import).

  knighton

  committed
  2a383b6

  Copy full SHA for 2a383b6