Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(iroh)!: remove tags from downloader #2348

Merged
merged 5 commits into from
Jun 7, 2024
Merged

Conversation

rklaehn
Copy link
Contributor

@rklaehn rklaehn commented Jun 5, 2024

Description

Remove tags from downloader. Adding temp or persistent tags is now the responsibility of the caller.

Breaking Changes

  • iroh_blobs::downloader::DownloadRequest
    • field tag is removed
    • fn untagged is removed
    • fn tag is removed

Docs are changed to indicate that tagging is now responibility of the caller...

Notes & open questions

Change checklist

  • Self-review.
  • Documentation updates if relevant.
  • Tests if relevant.
  • All breaking changes documented.

I saw that there is lots of logic related to tags in the downloader.

Do we really need this? One nice thing about temp tags or tags in general
is that you can assign them even before you have the data. So you can do
the whole tag and temp tag manager outside the downloader it seems.

Or am I missing somehting?
@rklaehn rklaehn requested a review from Frando June 5, 2024 12:51
@dignifiedquire
Copy link
Contributor

definitely looks like a nice simplification, but @Frando will have to answer if this is missing something crucial

@Frando
Copy link
Member

Frando commented Jun 7, 2024

Aha, interesting. I somehow did not realize that we can create tags before we have the data, I thought only temptags could do that. So I think this is fine. We should document this somewhere on the downloader though (it's public API) that users need to create tags themselves otherwise their newly downloaded blobs will be garbage collected at any time.

@Frando
Copy link
Member

Frando commented Jun 7, 2024

Now I remember something: The advantage of doing the tagging in the downloader is that if you queue the same downloader multiple times from the client API with SetTagOption::Auto you get only one auto tag. Whereas with this PR you'd get multiple auto tags. I think this was the original reason I moved the tagging into the downloader and added the TagSet.

@rklaehn
Copy link
Contributor Author

rklaehn commented Jun 7, 2024

One advantage of having it in the downloader is that if queuing the same downloader multiple times from the client API with SetTagOption::Auto you used to have only one auto tag, and now you'd get multiple auto tag. I think this was the original reason I moved the tagging into the downloader and added the TagSet.

Not sure that warrants the additional complexity. also, handing the same auto tag to 2 places is a bit dangerous, given that I think currently we are using auto tags as a workaround for the fact that temp tags are not available on the rpc client.

E.g. you got one place that does download with auto, then some validation, then removes the auto tag and replaces it with something permanent/named. The other side will not notice this, so they will have an auto tag that is not actually tagging anything anymore.

The assumption with an auto tag is that you own that tag, and handing out the same auto tag to 2 places means that you don't.

In general I think most uses for SetTagOption::Auto will be replaced by temp tags once we have them in the rpc client. We can make auto just some optional string prefix and a sufficiently large random number, then it does not even have to be part of the rpc api.

There is also the question whether to tag before or after. E.g. if you do a download you really care about you would assign a tag before you have the data to make sure the data is fully protected even against a crash/restart as it trickles in. There are so many ways to work with tags that I really think the downloader should not have anything to do with it and the job of protecting the data in some way should be up to the caller.

@rklaehn rklaehn marked this pull request as ready for review June 7, 2024 11:11
@dignifiedquire
Copy link
Contributor

please document the breaking changes

@dignifiedquire dignifiedquire changed the title refactor(iroh): remove tags from downloader refactor(iroh)!: remove tags from downloader Jun 7, 2024
@dignifiedquire dignifiedquire added this to the v0.18.0 milestone Jun 7, 2024
@rklaehn rklaehn force-pushed the rk/downloader-remove-tags branch from 3abfe6e to 969f3d4 Compare June 7, 2024 12:21
@rklaehn rklaehn added this pull request to the merge queue Jun 7, 2024
Merged via the queue into main with commit 82aa93f Jun 7, 2024
29 of 30 checks passed
@dignifiedquire dignifiedquire deleted the rk/downloader-remove-tags branch June 7, 2024 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants