Extend text to support yielding lines, paragraphs or documents

[albertvillanova]

Add config.row option to text module to allow yielding lines (default, current case), paragraphs or documents.

Feel free to comment on the name of the config parameter row:

• Currently, the docs state datasets are made of rows and columns
• Other names I considered: example, item

[lhoestq]

Awesome !

[lhoestq]

The parameter can also be named split_by with values "line", "paragraph" or "document" (no 's' at the end)

[albertvillanova]

The parameter can also be named split_by with values "line", "paragraph" or "document" (no 's' at the end)

@lhoestq @mariosasko I would avoid the term split in this context and keep it only for "train", "validation" and "test" splits.

• https://huggingface.co/docs/datasets/process.html#split

  datasets.Dataset.train_test_split() creates train and test splits, if your dataset doesn’t already have them.

• https://huggingface.co/docs/datasets/process.html#process-multiple-splits

  Many datasets have splits that you can process simultaneously with datasets.DatasetDict.map().

Please note that in the documentation, one of the terms more frequently used in this context is "row":

• https://huggingface.co/docs/datasets/access.html#features-and-columns

  A dataset is a table of rows and typed columns.

  Return the number of rows and columns with the following standard attributes:
  dataset.num_columns
  4
  dataset.num_rows
  3668

• https://huggingface.co/docs/datasets/access.html#rows-slices-batches-and-columns

  Get several rows of your dataset at a time with slice notation or a list of indices:

• https://huggingface.co/docs/datasets/process.html#map

  This function can even create new rows and columns.

Other of the terms more frequently used in the docs (in the code as well) is "example":

• https://huggingface.co/docs/datasets/process.html#map

  It allows you to apply a processing function to each example in a dataset, independently or in batches.

• https://huggingface.co/docs/datasets/process.html#batch-processing

  datasets.Dataset.map() also supports working with batches of examples.

• https://huggingface.co/docs/datasets/process.html#split-long-examples

  When your examples are too long, you may want to split them

• https://huggingface.co/docs/datasets/process.html#data-augmentation

  With batch processing, you can even augment your dataset with additional examples.

Less frequently used: "item":

• https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.add_item

  Add item to Dataset.

Other term used in the docs (although less frequently) is "sample". The advantage of this word is that it is also a verb, so we can use the parameter: "sample_by" (if you insist on using a verb instead of a noun).

In summary, these proposals:

• config.row
• config.example
• config.item
• config.sample
• config.sample_by

[lhoestq]

I like sample_by. Another idea I had was separate_by.

It could also be sampling, sampling_method, separation_method.

Not a big fan of the proposed nouns alone since they are very generic, that's why I tried to have something more specific.

I also agree that we actually should avoid split to avoid any confusion

[mariosasko]

Thanks for the analysis of the used terms. I also like sample_by (separate_by is good too).

[lhoestq]

Thank you !! :D