Skip to content
This repository has been archived by the owner on Jul 4, 2023. It is now read-only.

Allow StaticTokenizerEncoder to take any iterable #85

Closed
tbelhalfaoui opened this issue Nov 3, 2019 · 2 comments
Closed

Allow StaticTokenizerEncoder to take any iterable #85

tbelhalfaoui opened this issue Nov 3, 2019 · 2 comments

Comments

@tbelhalfaoui
Copy link

Actual Behavior

Right now, parameter sample of StaticTokenizerEncoder must be a list (explicit check).

It forces the user to pre-load the whole dataset in memory, which is not desirable for very large datasets.

Expected Behavior

It would be great if StaticTokenizerEncoder (and all child classes) could take any iterable for sample (not necessarily a list).

Therefore, sample could be for instance an iterator : the encoder would go once through the whole dataset to compute token counts, which could then be saved (e.g. pickled) for later use.
And token counts are typically much smaller than the dataset itself.

Steps to Reproduce the Problem

This raises a TypeError: Sample must be a list.

from torchnlp.encoders.text import WhitespaceEncoder
iterable = (x for x in ['hello world', 'PyTorch NLP'])
encoder = WhiteSpaceEncoder(iterable)

Proposal

This virtually just implies removing the explicit check (if not isinstance(sample, list) at torchnlp.encoders.text.StaticTokenizerEncoder:67).
I tried, and tests pass just fine. I can make a PR with this if you think this is a good idea.

@PetrochukM
Copy link
Owner

Hi There! I fixed this in #84. Thanks!

@tbelhalfaoui
Copy link
Author

Wow, that was quick. Thanks!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants