Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Break Stream in two, rewrite core part of Stream, in prep for new shards #547

Merged
merged 13 commits into from
Dec 25, 2023

Conversation

knighton
Copy link
Contributor

@knighton knighton commented Dec 24, 2023

Having parted the Streams:

  • The "upper" StreamCore is just all the args that are shared by all its Shards -- Shards then contain a reference to the owning StreamCore
  • The "lower" Stream (subclass) is everything else, nobody knows or cares StreamCore dirty circular import defeat trick exists
  • Also fix bug in generating deterministic temp path given remote + split when have remote but no local

Result of init sequence (more args to come):

StreamCore:
    remote: Optional[str]
    local: str
    split: Optional[str]
    download_retry: int
    download_timeout: float
    hash_algos: List[str]
    keep_old_phases: str
    safe_keep_old_phases: str
Stream:
    ... above plus ...
    proportion: Optional[float]
    repeat: Optional[float]
    choose: Optional[int]

@knighton knighton merged commit bdb5725 into dev Dec 25, 2023
@knighton knighton deleted the james/streamy branch December 25, 2023 01:08
karan6181 pushed a commit that referenced this pull request Jan 26, 2024
…rds (#547)

* Stream -> StreamCore (Shard args) + Stream (all).

* Drop pointless underscore vars.

* Auto keyword.

* Fix handling of generating local when default split.

* Clean up.

* Improve apply_defaults().

* Plug it in, propagate rewrites outward.

* Adjust keep_old_phases vs keep_zip handling.

* Adjust hash args handling.

* Default apply_defaults() args to auto so you don't have to provide them all.

* Update usage in test cases.

* Fix edge case.

* Another tweak.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant