Skip to content

Commit

Permalink
Fix loading a metric with internal import (#3077)
Browse files Browse the repository at this point in the history
* Test load metric with internal import

* Fix CanonicalMetricModuleFactory local imports
  • Loading branch information
albertvillanova authored Oct 14, 2021
1 parent f839338 commit 310a548
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 2 deletions.
2 changes: 1 addition & 1 deletion src/datasets/load.py
Original file line number Diff line number Diff line change
Expand Up @@ -581,7 +581,7 @@ def get_module(self) -> MetricModule:
imports = get_imports(local_path)
local_imports = _download_additional_modules(
name=self.name,
base_path=hf_github_url(path=self.name, name="", revision=revision),
base_path=hf_github_url(path=self.name, name="", revision=revision, dataset=False),
imports=imports,
download_config=self.download_config,
)
Expand Down
10 changes: 9 additions & 1 deletion tests/test_load.py
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,15 @@ def test_CanonicalDatasetModuleFactory(self):
module_factory_result = factory.get_module()
assert importlib.import_module(module_factory_result.module_path) is not None

def test_CanonicalMetricModuleFactory(self):
def test_CanonicalMetricModuleFactory_with_internal_import(self):
# "squad_v2" requires additional imports (internal)
factory = CanonicalMetricModuleFactory(
"squad_v2", download_config=self.download_config, dynamic_modules_path=self.dynamic_modules_path
)
module_factory_result = factory.get_module()
assert importlib.import_module(module_factory_result.module_path) is not None

def test_CanonicalMetricModuleFactory_with_external_import(self):
# "bleu" requires additional imports (external from github)
factory = CanonicalMetricModuleFactory(
"bleu", download_config=self.download_config, dynamic_modules_path=self.dynamic_modules_path
Expand Down

1 comment on commit 310a548

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Show benchmarks

PyArrow==3.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric read_batch_formatted_as_numpy after write_array2d read_batch_formatted_as_numpy after write_flattened_sequence read_batch_formatted_as_numpy after write_nested_sequence read_batch_unformated after write_array2d read_batch_unformated after write_flattened_sequence read_batch_unformated after write_nested_sequence read_col_formatted_as_numpy after write_array2d read_col_formatted_as_numpy after write_flattened_sequence read_col_formatted_as_numpy after write_nested_sequence read_col_unformated after write_array2d read_col_unformated after write_flattened_sequence read_col_unformated after write_nested_sequence read_formatted_as_numpy after write_array2d read_formatted_as_numpy after write_flattened_sequence read_formatted_as_numpy after write_nested_sequence read_unformated after write_array2d read_unformated after write_flattened_sequence read_unformated after write_nested_sequence write_array2d write_flattened_sequence write_nested_sequence
new / old (diff) 0.010451 / 0.011353 (-0.000902) 0.004554 / 0.011008 (-0.006454) 0.037194 / 0.038508 (-0.001315) 0.036732 / 0.023109 (0.013623) 0.389853 / 0.275898 (0.113955) 0.393977 / 0.323480 (0.070497) 0.007312 / 0.007986 (-0.000674) 0.005505 / 0.004328 (0.001176) 0.011504 / 0.004250 (0.007253) 0.041726 / 0.037052 (0.004674) 0.381622 / 0.258489 (0.123133) 0.375481 / 0.293841 (0.081640) 0.035592 / 0.128546 (-0.092954) 0.013035 / 0.075646 (-0.062611) 0.316893 / 0.419271 (-0.102379) 0.055110 / 0.043533 (0.011577) 0.353571 / 0.255139 (0.098432) 0.386630 / 0.283200 (0.103430) 0.083674 / 0.141683 (-0.058009) 1.896552 / 1.452155 (0.444397) 2.032751 / 1.492716 (0.540035)

Benchmark: benchmark_getitem_100B.json

metric get_batch_of_1024_random_rows get_batch_of_1024_rows get_first_row get_last_row
new / old (diff) 0.239059 / 0.018006 (0.221052) 0.544071 / 0.000490 (0.543581) 0.012654 / 0.000200 (0.012454) 0.000348 / 0.000054 (0.000294)

Benchmark: benchmark_indices_mapping.json

metric select shard shuffle sort train_test_split
new / old (diff) 0.045211 / 0.037411 (0.007800) 0.026777 / 0.014526 (0.012251) 0.028685 / 0.176557 (-0.147871) 0.140381 / 0.737135 (-0.596755) 0.030630 / 0.296338 (-0.265709)

Benchmark: benchmark_iterating.json

metric read 5000 read 50000 read_batch 50000 10 read_batch 50000 100 read_batch 50000 1000 read_formatted numpy 5000 read_formatted pandas 5000 read_formatted tensorflow 5000 read_formatted torch 5000 read_formatted_batch numpy 5000 10 read_formatted_batch numpy 5000 1000 shuffled read 5000 shuffled read 50000 shuffled read_batch 50000 10 shuffled read_batch 50000 100 shuffled read_batch 50000 1000 shuffled read_formatted numpy 5000 shuffled read_formatted_batch numpy 5000 10 shuffled read_formatted_batch numpy 5000 1000
new / old (diff) 0.595613 / 0.215209 (0.380404) 5.849264 / 2.077655 (3.771610) 2.239823 / 1.504120 (0.735703) 1.907044 / 1.541195 (0.365850) 1.892274 / 1.468490 (0.423784) 0.590145 / 4.584777 (-3.994632) 6.643594 / 3.745712 (2.897882) 1.410305 / 5.269862 (-3.859557) 1.284588 / 4.565676 (-3.281088) 0.063725 / 0.424275 (-0.360550) 0.005167 / 0.007607 (-0.002441) 0.755563 / 0.226044 (0.529518) 7.434790 / 2.268929 (5.165862) 3.026688 / 55.444624 (-52.417937) 2.316875 / 6.876477 (-4.559602) 2.314494 / 2.142072 (0.172422) 0.746142 / 4.805227 (-4.059086) 0.140966 / 6.500664 (-6.359698) 0.055165 / 0.075469 (-0.020304)

Benchmark: benchmark_map_filter.json

metric filter map fast-tokenizer batched map identity map identity batched map no-op batched map no-op batched numpy map no-op batched pandas map no-op batched pytorch map no-op batched tensorflow
new / old (diff) 1.771721 / 1.841788 (-0.070066) 14.419279 / 8.074308 (6.344971) 39.429877 / 10.191392 (29.238485) 0.919695 / 0.680424 (0.239271) 0.681277 / 0.534201 (0.147076) 0.294871 / 0.579283 (-0.284412) 0.752020 / 0.434364 (0.317656) 0.243426 / 0.540337 (-0.296911) 0.249857 / 1.386936 (-1.137080)
PyArrow==latest
Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric read_batch_formatted_as_numpy after write_array2d read_batch_formatted_as_numpy after write_flattened_sequence read_batch_formatted_as_numpy after write_nested_sequence read_batch_unformated after write_array2d read_batch_unformated after write_flattened_sequence read_batch_unformated after write_nested_sequence read_col_formatted_as_numpy after write_array2d read_col_formatted_as_numpy after write_flattened_sequence read_col_formatted_as_numpy after write_nested_sequence read_col_unformated after write_array2d read_col_unformated after write_flattened_sequence read_col_unformated after write_nested_sequence read_formatted_as_numpy after write_array2d read_formatted_as_numpy after write_flattened_sequence read_formatted_as_numpy after write_nested_sequence read_unformated after write_array2d read_unformated after write_flattened_sequence read_unformated after write_nested_sequence write_array2d write_flattened_sequence write_nested_sequence
new / old (diff) 0.011986 / 0.011353 (0.000633) 0.004623 / 0.011008 (-0.006385) 0.039310 / 0.038508 (0.000802) 0.040281 / 0.023109 (0.017172) 0.377811 / 0.275898 (0.101913) 0.425918 / 0.323480 (0.102439) 0.010092 / 0.007986 (0.002106) 0.005258 / 0.004328 (0.000930) 0.010513 / 0.004250 (0.006263) 0.048408 / 0.037052 (0.011355) 0.375721 / 0.258489 (0.117232) 0.466255 / 0.293841 (0.172414) 0.038527 / 0.128546 (-0.090020) 0.012522 / 0.075646 (-0.063124) 0.327080 / 0.419271 (-0.092192) 0.061216 / 0.043533 (0.017683) 0.391066 / 0.255139 (0.135927) 0.427579 / 0.283200 (0.144379) 0.098222 / 0.141683 (-0.043461) 2.153246 / 1.452155 (0.701092) 2.317569 / 1.492716 (0.824853)

Benchmark: benchmark_getitem_100B.json

metric get_batch_of_1024_random_rows get_batch_of_1024_rows get_first_row get_last_row
new / old (diff) 0.377705 / 0.018006 (0.359699) 0.540694 / 0.000490 (0.540204) 0.076653 / 0.000200 (0.076453) 0.000830 / 0.000054 (0.000775)

Benchmark: benchmark_indices_mapping.json

metric select shard shuffle sort train_test_split
new / old (diff) 0.038966 / 0.037411 (0.001555) 0.025988 / 0.014526 (0.011462) 0.029534 / 0.176557 (-0.147023) 0.141117 / 0.737135 (-0.596018) 0.032051 / 0.296338 (-0.264287)

Benchmark: benchmark_iterating.json

metric read 5000 read 50000 read_batch 50000 10 read_batch 50000 100 read_batch 50000 1000 read_formatted numpy 5000 read_formatted pandas 5000 read_formatted tensorflow 5000 read_formatted torch 5000 read_formatted_batch numpy 5000 10 read_formatted_batch numpy 5000 1000 shuffled read 5000 shuffled read 50000 shuffled read_batch 50000 10 shuffled read_batch 50000 100 shuffled read_batch 50000 1000 shuffled read_formatted numpy 5000 shuffled read_formatted_batch numpy 5000 10 shuffled read_formatted_batch numpy 5000 1000
new / old (diff) 0.640727 / 0.215209 (0.425518) 5.999679 / 2.077655 (3.922024) 2.373340 / 1.504120 (0.869220) 2.066564 / 1.541195 (0.525369) 1.999309 / 1.468490 (0.530819) 0.637605 / 4.584777 (-3.947172) 6.765660 / 3.745712 (3.019948) 1.434905 / 5.269862 (-3.834957) 1.365169 / 4.565676 (-3.200507) 0.064789 / 0.424275 (-0.359486) 0.006112 / 0.007607 (-0.001495) 0.732399 / 0.226044 (0.506355) 7.582466 / 2.268929 (5.313537) 3.147524 / 55.444624 (-52.297100) 2.366546 / 6.876477 (-4.509931) 2.340107 / 2.142072 (0.198034) 0.804949 / 4.805227 (-4.000278) 0.149097 / 6.500664 (-6.351567) 0.058109 / 0.075469 (-0.017360)

Benchmark: benchmark_map_filter.json

metric filter map fast-tokenizer batched map identity map identity batched map no-op batched map no-op batched numpy map no-op batched pandas map no-op batched pytorch map no-op batched tensorflow
new / old (diff) 1.867882 / 1.841788 (0.026094) 15.479283 / 8.074308 (7.404975) 43.230091 / 10.191392 (33.038699) 0.965948 / 0.680424 (0.285524) 0.676583 / 0.534201 (0.142382) 0.290118 / 0.579283 (-0.289165) 0.728091 / 0.434364 (0.293727) 0.249498 / 0.540337 (-0.290839) 0.277607 / 1.386936 (-1.109329)

CML watermark

Please sign in to comment.