Test Analysis

To enable test analysis:

via commandline use --test-reliability=true

via configuration: in appsettings.json, set the property HistoryAnalysisConfiguration => Enabled = true

Description

Test analysis keeps a history of your test runs in a local database history file. By default it will save your last 50 test runs. At the end of each test run it will compare the current test run to your history and look for potential issues you might want to be warned about which will appear in your test run report.

You won't see any output until at least 5 test runs have been collected, or the configuration value MinTestHistoryToAnalyze you have set it to.

Configuration

Located in appsettings.json in your NUnit.Commander installation folder.

/// <summary>
/// True to enable history analysis
/// </summary>
bool Enabled { get; set; }

/// <summary>
/// Specify the number of test runs to store for reliability analysis
/// </summary>
int MaxTestReliabilityRuns { get; set; } = 50;

/// <summary>
/// The minimum number of history entries to analyze
/// </summary>
int MinTestHistoryToAnalyze { get; set; } = 5;

/// <summary>
/// The minimum percentage (0.001-1.0) of failed tests allowed. Default: 0.05 (5%)
/// </summary>
[Range(0.001, 1.0)]
double MinTestReliabilityThreshold { get; set; } = 0.05;

/// <summary>
/// The maximum percentage (0.001-1.0) of a tests duration can change before triggering failure. Default: 0.4 (40%)
/// </summary>
[Range(0.001, 1.0)]
double MaxTestDurationChange { get; set; } = 0.4;

/// <summary>
/// Tests that complete less than this time will not have duration analysis checked
/// </summary>
int MinTestMillisecondsForDurationAnalysis { get; set; } = 1000;