Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

merge #5

Merged
merged 56 commits into from
Apr 16, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
8f1fd10
refactor: downloader
Frando Feb 9, 2024
99db6ef
test: add test for concurrent progress reporting
Frando Mar 15, 2024
98fa4fc
chore: cleanup
Frando Mar 15, 2024
4fc6568
refactor: rename some things for clarity
Frando Mar 16, 2024
5090058
feat: limit concurrent dials per download
Frando Mar 17, 2024
32e6cca
chore: cleanup
Frando Mar 17, 2024
6064e73
Merge remote-tracking branch 'origin/main' into feat/downloader-next
Frando Mar 19, 2024
6c76517
refactor: respect progress sender IDs to allow reusing the senders in…
Frando Mar 19, 2024
2fbd433
chore: fmt & clippy
Frando Mar 19, 2024
91f436e
cleanup: shared download progress
Frando Mar 20, 2024
bdf513a
feat: handle tags in the downloader
Frando Mar 20, 2024
831adcb
Merge remote-tracking branch 'origin/main' into feat/downloader-next
Frando Mar 20, 2024
699b009
fix: visibility
Frando Mar 20, 2024
ad3dd0e
fixup
Frando Mar 20, 2024
cf73c57
fixup
Frando Mar 20, 2024
24ae6a3
chore: clippy
Frando Mar 20, 2024
7641ac6
fixup
Frando Mar 20, 2024
a54078a
Merge remote-tracking branch 'origin/main' into feat/downloader-next
Frando Mar 22, 2024
666af33
Merge remote-tracking branch 'origin/main' into feat/downloader-next
Frando Mar 22, 2024
5733460
refactor: make export a seperate operation from download
Frando Mar 22, 2024
2e989db
fixup after merge
Frando Mar 22, 2024
d1a37c0
refactor: remove id mapping
Frando Mar 22, 2024
e40b1e6
refactor: move BlobId up to the progress events
Frando Mar 22, 2024
6381a51
fix tests
Frando Mar 22, 2024
ea8c6e8
Merge remote-tracking branch 'origin/main' into feat/downloader-next
Frando Mar 22, 2024
6946534
Merge remote-tracking branch 'origin/main' into feat/downloader-next
Frando Mar 26, 2024
0bfe6a3
cleanup
Frando Mar 27, 2024
15c0d30
fix: invariants
Frando Mar 27, 2024
d89349e
fixes and cleanups
Frando Mar 27, 2024
40d9fea
fix: nodes_have should only add to queued downloads
Frando Mar 27, 2024
b33ce55
chore: fmt
Frando Mar 27, 2024
4aaf7b4
chore: clippy!
Frando Mar 27, 2024
7b8b587
fix: do not queue downloads with no providers
Frando Mar 27, 2024
cbaa794
feat: set more than one node on download requests (#2128)
Frando Apr 2, 2024
146c10a
fix: address review by divma
Frando Apr 4, 2024
f82c812
fix: remove now obsolete into_send method from ProgressSender
Frando Apr 4, 2024
de09ab4
chore: remove unused code
Frando Apr 4, 2024
f0eb396
Merge branch 'main' into feat/downloader-next
Frando Apr 4, 2024
bfd2a47
fix: use Display for DownloadKind
Frando Apr 8, 2024
1a269de
fix: first round of changes after review
Frando Apr 8, 2024
5fcc671
docs: downloader next_step
Frando Apr 8, 2024
f634d79
more review
Frando Apr 8, 2024
848ee46
more review
Frando Apr 8, 2024
d0fae08
fix: only remove hashes from provider map if they are really not needed
Frando Apr 8, 2024
6b7d3fb
refactor: move TagSet into util
Frando Apr 8, 2024
904a3d3
refactor: store intents in request info
Frando Apr 8, 2024
1629548
Merge remote-tracking branch 'origin/main' into feat/downloader-next
Frando Apr 8, 2024
b86dbe9
cleanup
Frando Apr 8, 2024
393b7de
refactor: no allocation in next_step and simpler code
Frando Apr 8, 2024
91db7ab
Merge remote-tracking branch 'origin/main' into feat/downloader-next
Frando Apr 8, 2024
c8b6af7
fix: test after merge
Frando Apr 9, 2024
643fb6e
Merge remote-tracking branch 'origin/main' into feat/downloader-next
Frando Apr 9, 2024
84a16e1
refactor: better method
Frando Apr 9, 2024
1e84ae0
fix(iroh-bytes): do not log redundant file delete error (#2199)
rklaehn Apr 16, 2024
2bb7bd5
fix(iroh-dns-server): fix bug in pkarr name parsing (#2200)
Frando Apr 16, 2024
0352e9c
Merge remote-tracking branch 'origin/main' into feat/downloader-next
Frando Apr 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

28 changes: 25 additions & 3 deletions iroh-base/src/hash.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ use bao_tree::blake3;
use postcard::experimental::max_size::MaxSize;
use serde::{de, Deserialize, Deserializer, Serialize, Serializer};

use crate::base32::{parse_array_hex_or_base32, HexOrBase32ParseError};
use crate::base32::{self, parse_array_hex_or_base32, HexOrBase32ParseError};

/// Hash type used throughout.
#[derive(PartialEq, Eq, Copy, Clone, Hash)]
Expand Down Expand Up @@ -54,6 +54,12 @@ impl Hash {
pub fn to_hex(&self) -> String {
self.0.to_hex().to_string()
}

/// Convert to a base32 string limited to the first 10 bytes for a friendly string
/// representation of the hash.
pub fn fmt_short(&self) -> String {
base32::fmt_short(self.as_bytes())
}
}

impl AsRef<[u8]> for Hash {
Expand Down Expand Up @@ -173,7 +179,18 @@ impl MaxSize for Hash {

/// A format identifier
#[derive(
Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize, Default, Debug, MaxSize,
Clone,
Copy,
PartialEq,
Eq,
PartialOrd,
Ord,
Serialize,
Deserialize,
Default,
Debug,
MaxSize,
Hash,
)]
pub enum BlobFormat {
/// Raw blob
Expand Down Expand Up @@ -205,7 +222,7 @@ impl BlobFormat {
}

/// A hash and format pair
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, MaxSize)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, MaxSize, Hash)]
pub struct HashAndFormat {
/// The hash
pub hash: Hash,
Expand Down Expand Up @@ -289,6 +306,11 @@ mod redb_support {
}

impl HashAndFormat {
/// Create a new hash and format pair.
pub fn new(hash: Hash, format: BlobFormat) -> Self {
Self { hash, format }
}

/// Create a new hash and format pair, using the default (raw) format.
pub fn raw(hash: Hash) -> Self {
Self {
Expand Down
4 changes: 3 additions & 1 deletion iroh-bytes/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ flume = "0.11"
futures = "0.3.25"
futures-buffered = "0.2.4"
genawaiter = { version = "0.99.1", features = ["futures03"] }
hashlink = { version = "0.9.0", optional = true }
hex = "0.4.3"
iroh-base = { version = "0.14.0", features = ["redb"], path = "../iroh-base" }
iroh-io = { version = "0.6.0", features = ["stats"] }
Expand All @@ -51,6 +52,7 @@ tracing-futures = "0.2.5"

[dev-dependencies]
http-body = "0.4.5"
iroh-bytes = { path = ".", features = ["downloader"] }
iroh-test = { path = "../iroh-test" }
proptest = "1.0.0"
serde_json = "1.0.107"
Expand All @@ -63,8 +65,8 @@ tempfile = "3.10.0"

[features]
default = ["fs-store"]
downloader = ["iroh-net", "parking_lot", "tokio-util/time", "hashlink"]
fs-store = ["reflink-copy", "redb", "redb_v1", "tempfile"]
downloader = ["iroh-net", "parking_lot", "tokio-util/time"]
metrics = ["iroh-metrics"]

[[example]]
Expand Down
Loading
Loading