Turn PackageValidation on by default

[safern]

In .NET 6 we added a new tool to help library developers author their packages without breaking changes by raising errors after the Pack target is executed. Currently that tool is opt-in, which means authors of libraries, need to set <EnablePackageValidation>true</EnablePackageValidation> in their csproj's.

In order to turn it on by default we need to investigate/invest in different areas:

Performance

• [Package Validation] investigate performance impact to the developer innerloop when creating a package #23893
• [Package Validation] Explore a better way to examine the package for better performance #23896
• PackageValidation runs in no-op incremental builds #23517

Diagnostic experience

Do we have a good error experience?

We should evaluate if we can improve this experience by making errors more actionable maybe with adding support to suppress them via the IDE. If someone creates the package today from Visual Studio, they would get the errors, however, to suppress them if they are intentional, they need to run dotnet pack /p:GenerateCompatibilitySuppressionFile=true; which means having to leave the IDE.

Is the suppressions file readable?

• [Package Validation] Link to diagnostics ID docs page in suppression files #19515

Compatibility Rules

Currently if we were to turn package validation on by default, all our costumers would get errors if any, and they would start discovering this tool, however, we don't have a complete set of rules to cover all breaking changes, that can compromise our costumers trust, because they could ship a package thinking that they are fully compatible because their pack target was clean. However, with the gaps of rules we would be lying as a breaking change could be introduced and not caught because a rule is missing.

Current list of implemented rules

• Types must exist
• Members must exist (this covers adding an optional parameter to a current member)
• Assembly identity should match
• Can not add abstract member to unsealed type
• Can not add member to interface without default implementation
• Can not remove base type or interface from hierarchy tree
• Can not seal type

Missing Rules

• [API Compat] Add rule to make sure nullable annotations are compatible #18681
• [API Compat] Add rule to make sure virtual modifier is not removed #18682
• [API Compat] Add rule to prohibit making a member virtual #18683
• [API Compat] Add rule to make sure that enum underlying type matches #23902
• [API Compat] add rule to make sure enum values match #23903
• [API Compat] add rule to make sure that a member is not made more visible #23904
• [API Compat] add rule that ensures there is not attribute differences in between sides #23905
• [API Compat] add rule that makes sure that type classifications are not changed #23906
• [API Compat] add rule to make sure that parameter and return type modifiers don't change #23907
• [API Compat] add rule to make sure parameter names don't change #23908
• [API Compat] add rule that make sure that delegates match #23909

Some of the missing rules are dependent on having references loaded as part of the compilation, so maybe completion of this issue is needed as well:

• [API Compat] add rule that make sure that delegates match #23909

That scenario is already covered for comparing assemblies within the same package, however we didn't enable it for package baseline validation because there was not a good way to get the dependencies for the package baseline since we are using package download and we can't get dependencies for it. This would be something we can collaborate with NuGet to improve the experience of baseline validation.

Support Multiple Assemblies On Packages

Currently, we just focused on the simple scenario that a package contained only a single assembly and the assembly was named the same as the package id. However, this is not true for packages out there, a package can have multiple assemblies that have different names, and we should ensure we run validation for those assemblies.

• [Package Validation] Add support for multiple assemblies in the package. #17364

Support automatically getting latest released package

For package baseline validation we should explore the option of supporting fetching the latest stable released package to run baseline validation again. Also, there is a first party request to support wildcards or some form of range to automatically get the baseline package version respecting that range:

• [Package Validation] Add an opt-in feature to automatically calculate the package to download for API comparison #25268

cc: @joperezr @ericstj