v2.5.0

[epwalsh]

🆕 AllenNLP v2.5.0 comes with a few big new features and improvements 🆕

There is a whole new module allennlp.fairness that contains implementations of fairness metrics, bias metrics, and bias mitigation tools for your models thanks to @ArjunSubramonian. For a great introduction, check out the corresponding chapter of the guide: https://guide.allennlp.org/fairness.

Another major addition is the allennlp.confidence_checks.task_checklists submodule, thanks to @AkshitaB, which provides an automated way to run behavioral tests of your models using the checklist library.

BeamSearch also has several new important features, including an easy to add arbitrary constraints, thanks to @danieldeutsch.

See below for a comprehensive list of updates 👇

What's new

Added 🎉

• Added TaskSuite base class and command line functionality for running checklist test suites, along with implementations for SentimentAnalysisSuite, QuestionAnsweringSuite, and TextualEntailmentSuite. These can be found in the allennlp.confidence_checks.task_checklists module.
• Added BiasMitigatorApplicator, which wraps any Model and mitigates biases by finetuning
  on a downstream task.
• Added allennlp diff command to compute a diff on model checkpoints, analogous to what git diff does on two files.
• Meta data defined by the class allennlp.common.meta.Meta is now saved in the serialization directory and archive file
  when training models from the command line. This is also now part of the Archive named tuple that's returned from load_archive().
• Added nn.util.distributed_device() helper function.
• Added allennlp.nn.util.load_state_dict helper function.
• Added a way to avoid downloading and loading pretrained weights in modules that wrap transformers
  such as the PretrainedTransformerEmbedder and PretrainedTransformerMismatchedEmbedder.
  You can do this by setting the parameter load_weights to False.
  See PR #5172 for more details.
• Added SpanExtractorWithSpanWidthEmbedding, putting specific span embedding computations into the _embed_spans method and leaving the common code in SpanExtractorWithSpanWidthEmbedding to unify the arguments, and modified BidirectionalEndpointSpanExtractor, EndpointSpanExtractor and SelfAttentiveSpanExtractor accordingly. Now, SelfAttentiveSpanExtractor can also embed span widths.
• Added a min_steps parameter to BeamSearch to set a minimum length for the predicted sequences.
• Added the FinalSequenceScorer abstraction to calculate the final scores of the generated sequences in BeamSearch.
• Added shuffle argument to BucketBatchSampler which allows for disabling shuffling.
• Added allennlp.modules.transformer.attention_module which contains a generalized AttentionModule. SelfAttention and T5Attention both inherit from this.
• Added a Constraint abstract class to BeamSearch, which allows for incorporating constraints on the predictions found by BeamSearch,
  along with a RepeatedNGramBlockingConstraint constraint implementation, which allows for preventing repeated n-grams in the output from BeamSearch.
• Added DataCollator for dynamic operations for each batch.

Changed ⚠️

• Use dist_reduce_sum in distributed metrics.
• Allow Google Cloud Storage paths in cached_path ("gs://...").
• Renamed nn.util.load_state_dict() to read_state_dict to avoid confusion with torch.nn.Module.load_state_dict().
• TransformerModule.from_pretrained_module now only accepts a pretrained model ID (e.g. "bert-base-case") instead of
  an actual torch.nn.Module. Other parameters to this method have changed as well.
• Print the first batch to the console by default.
• Renamed sanity_checks to confidence_checks (sanity_checks is deprecated and will be removed in AllenNLP 3.0).
• Trainer callbacks can now store and restore state in case a training run gets interrupted.
• VilBERT backbone now rolls and unrolls extra dimensions to handle input with > 3 dimensions.
• BeamSearch is now a Registrable class.

Fixed ✅

• When PretrainedTransformerIndexer folds long sequences, it no longer loses the information from token type ids.
• Fixed documentation for GradientDescentTrainer.cuda_device.
• Re-starting a training run from a checkpoint in the middle of an epoch now works correctly.
• When using the "moving average" weights smoothing feature of the trainer, training checkpoints would also get smoothed, with strange results for resuming a training job. This has been fixed.
• When re-starting an interrupted training job, the trainer will now read out the data loader even for epochs and batches that can be skipped. We do this to try to get any random number generators used by the reader or data loader into the same state as they were the first time the training job ran.
• Fixed the potential for a race condition with cached_path() when extracting archives. Although the race condition
  is still possible if used with force_extract=True.
• Fixed wandb callback to work in distributed training.
• Fixed tqdm logging into multiple files with allennlp-optuna.

Commits

b92fd9a Contextualized bias mitigation (#5176)
aa52a9a Checklist fixes (#5239)
6206797 Fix tqdm logging into multiple files with allennlp-optuna (#5235)
b0aa1d4 Generalize T5 modules (#5166)
5b111d0 tick version for nightly release
39d7e5a Make BeamSearch Registrable (#5231)
c014232 Add constraints to beam search (#5216)
98dae7f Emergency fix. I forgot to take this out.
c5bff8b Fixes Checkpointing (#5220)
3d5799d Roll backbone (#5229)
babc450 Added DataCollator for dynamic operations for each batch. (#5221)
d97ed40 Bump checklist from 0.0.10 to 0.0.11 (#5222)
12155c4 fix race condition when extracting files with cached_path (#5227)
d662977 cancel redundant GH Actions workflows (#5226)
2d8f390 Fix W&B callback for distributed training (#5223)
59df2ad Update nr-interface requirement from <0.0.4 to <0.0.6 (#5213)
3e1b553 Bump black from 20.8b1 to 21.5b1 (#5195)
d2840cb save meta data with model archives (#5209)
bd941c6 added shuffle disable option in BucketBatchSampler (#5212)
3585c9f Implementing abstraction to score final sequences in BeamSearch (#5208)
79d16af Add a min_steps parameter to BeamSearch (#5207)
cf113d7 Changes and improvements to how we initialize transformer modules from pretrained models (#5200)
cccb35d Rename sanity_checks to confidence_checks (#5201)
db8ff67 Update transformers requirement from <4.6,>=4.1 to >=4.1,<4.7 (#5199)
fd5c9e4 Bias Metrics (#5139)
d9b19b6 Bias Mitigation and Direction Methods (#5130)
7473737 add diff command (#5109)
d85c5c3 Explicitly pass serialization directory and local rank to trainer in train command (#5180)
96c3caf fix nltk downloads in install (#5189)
b1b455a improve contributing guide / PR template (#5185)
7a260da fix cuda_device docs (#5188)
0bf590d Update Makefile (#5183)
3335700 Default print first batch (#5175)
b533733 Refactor span extractors and unify forward. (#5160)
01b232f Allow google cloud storage locations for cached_path (#5173)
eb2ae30 Update README.md (#5165)
55efa68 fix dataclasses import (#5169)
a463e0e Add way of skipping pretrained weights download (#5172)
c71bb46 improve err msg for PolynomialDecay LR scheduler (#5143)
530dae4 Simplify metrics (#5154)
12f5b0f Run some slow tests on the self-hosted runner (#5161)
9091580 Fixes token type ids for folded sequences (#5149)
10400e0 Run checklist suites in AllenNLP (#5065)
d11359e make dist_reduce_sum work for tensors (#5147)
9184fbc Fixes Backbone / Model MRO inconsistency (#5148)

---

This discussion was created from the release v2.5.0.