Introduction of 'MustPassRepeatedly' decorator (#1051)

* introduction of RepeatAttempts decorator

* first review

* Refactor: renamed 'test' to 'spec' in various parts of the doc and code

* remove repeated FAILEd log; correct entry point to FLAKEY TEST log

* removing Focus from test

* remove multipleExecutionDecorator type switch

* remove multipleExecutionDecorator type switch - pt 2

decorator_dsl.go

@@ -13,13 +13,21 @@ You can learn more about decorators here: https://onsi.github.io/ginkgo/#decorat
 type Offset = internal.Offset
 
 /*
-FlakeAttempts(uint N) is a decorator that allows you to mark individual specs or spec containers as flaky.  Ginkgo will run them up to `N` times until they pass.
+FlakeAttempts(uint N) is a decorator that allows you to mark individual specs or spec containers as flaky. Ginkgo will run them up to `N` times until they pass.
 
-You can learn more here: https://onsi.github.io/ginkgo/#repeating-spec-runs-and-managing-flaky-specs
+You can learn more here: https://onsi.github.io/ginkgo/#the-flakeattempts-decorator
 You can learn more about decorators here: https://onsi.github.io/ginkgo/#decorator-reference
 */
 type FlakeAttempts = internal.FlakeAttempts
 
+/*
+MustPassRepeatedly(uint N) is a decorator that allows you to repeat the execution of individual specs or spec containers. Ginkgo will run them up to `N` times until they fail.
+
+You can learn more here: https://onsi.github.io/ginkgo/#the-mustpassrepeatedly-decorator
+You can learn more about decorators here: https://onsi.github.io/ginkgo/#decorator-reference
+*/
+type MustPassRepeatedly = internal.MustPassRepeatedly
+
 /*
 Focus is a decorator that allows you to mark a spec or container as focused.  Identical to FIt and FDescribe.
 

docs/index.md

@@ -2487,7 +2487,21 @@ One quick note on `--repeat`: when you invoke `ginkgo --repeat=N` Ginkgo will ru
 
 Both `--until-it-fails` and `--repeat` help you identify flaky specs early.  Doing so will help you debug flaky specs while the context that introduced them is fresh.
 
-However.  There are times when the cost of preventing and/or debugging flaky specs simply is simply too high and specs simply need to be retried.  While this should never be the primary way of dealing with flaky specs, Ginkgo is pragmatic about this reality and provides a mechanism for retrying specs.
+A more granular approach to repeating specs is by decorating individual subject or container nodes with the MustPassRepeatedly(N) decorator:
+
+```go
+Describe("Storing books", func() {
+  It("can save books to the central library", MustPassRepeatedly(3), func() {
+    // this spec has been marked and will be retried up to 3 times
+  })
+
+  It("can save books locally", func() {
+    // this spec has not been marked and will not be retired
+  })
+})
+```
+
+However,  There are times when the cost of preventing and/or debugging flaky specs simply is simply too high and specs simply need to be retried.  While this should never be the primary way of dealing with flaky specs, Ginkgo is pragmatic about this reality and provides a mechanism for retrying specs.
 
 You can retry all specs in a suite via:
 
@@ -2497,7 +2511,7 @@ ginkgo --flake-attempts=N
 
 Now, when a spec fails Ginkgo will not automatically mark the suite as failed.  Instead it will attempt to rerun the spec up to `N` times.  If the spec succeeds during a retry, Ginkgo moves on and marks the suite as successful but reports that the spec needed to be retried.
 
-You can take a more granular approach by decorating individual subject nodes or container nodes as potentially flaky with the `FlakeAttempts(N)` decorator:
+A more granular approach is also provided for this functionality with the use of the `FlakeAttempts(N)` decorator:
 
 ```go
 Describe("Storing books", func() {
@@ -4799,9 +4813,9 @@ In addition to `Offset`, users can decorate nodes with a `types.CodeLocation`.
 Passing a `types.CodeLocation` decorator in has the same semantics as passing `Offset` in: it only applies to the node in question.
 
 #### The FlakeAttempts Decorator
-The `FlakeAttempts(uint)` decorator applies container and subject nodes.  It is an error to apply `FlakeAttempts` to a setup node.
+The `FlakeAttempts(uint)` decorator applies to container and subject nodes.  It is an error to apply `FlakeAttempts` to a setup node.
 
-`FlakeAttempts` allows the user to flag specific tests or groups of tests as potentially flaky.  Ginkgo will run tests up to the number of times specified in `FlakeAttempts` until they pass.  For example:
+`FlakeAttempts` allows the user to flag specs trees as potentially flaky.  Ginkgo will retry the spec up to the number of times specified in `FlakeAttempts` until they pass.  For example:
 
 ```go
 Describe("flaky tests", FlakeAttempts(3), func() {
@@ -4825,8 +4839,38 @@ Describe("flaky tests", FlakeAttempts(3), func() {
 
 With this setup, `"is flaky"` and `"is also flaky"` will run up to 3 times.  `"is _really_ flaky"` will run up to 5 times.  `"is _not_ flaky"` will run only once.  Note that if multiple `FlakeAttempts` appear in a spec's hierarchy, the most deeply nested `FlakeAttempts` wins.  If multiple `FlakeAttempts` are passed into a given node, the last one wins.
 
-If `ginkgo --flake-attempts=N` is set the value passed in by the CLI will override all the decorated values.  Every test will now run up to `N` times.
+If `ginkgo --flake-attempts=N` is set the value passed in by the CLI will override all the decorated values.  Every spec in the test suite will now run up to `N` times.
+
+#### The MustPassRepeatedly Decorator
+The `MustPassRepeatedly(uint)` decorator applies to container and subject nodes.  It is an error to apply `MustPassRepeatedly` to a setup node.
+
+the `MustPassRepeatedly` flag allows the user to repeatedly run specs in a controlled manner.  Ginkgo will repeatedly run specs up to the number of times specified in `MustPassRepeatedly` or until they fail.  For example:
+
+```go
+Describe("repeated specs", MustPassRepeatedly(3), func() {
+  It("is repeated", func() {
+    ...
+  })
+
+  It("is also repeated", func() {
+    ...
+  })
+
+  It("is repeated even more", MustPassRepeatedly(5) func() {
+    ...
+  })
+
+  It("is repeated less", MustPassRepeatedly(1), func() {
+    ...
+  })
+})
+```
+
+With this setup, `"is repeated"` and `"is also repeated"` will run up to 3 times.  `"is repeated even more"` will run up to 5 times.  `"is repeated less"` will run only once.  Note that if multiple `MustPassRepeatedly` appear in a spec's hierarchy, the most deeply nested `MustPassRepeatedly` wins.  If multiple `MustPassRepeatedly` are passed into a given node, the last one wins.
+
+The `ginkgo --repeat=N` value passed in by the CLI has no relation with the `MustPassRepeatedly` decorator. If the `--repeat` CLI flag is used and a container or subject node also contains the `MustPassRepeatedly` decorator, then the spec will run up to `N*R` times, where `N` is the values passed to the `--repeat` CLI flag and `R` is the value passed to the MustPassRepeatedly decorator.
 
+If the `MustPassRepeatedly` decorator is set, it will override the `ginkgo --flake-attempts=N` CLI config. The specs that do not contain the `MustPassRepeatedly(R)` decorator will still run up to `N` times, in accordance to the `ginkgo --flake-attempts=N` CLI config.
 
 #### The SuppressProgressOutput Decorator
 

dsl/decorators/decorators_dsl.go

@@ -1,7 +1,7 @@
 /*
 Ginkgo is usually dot-imported via:
 
-    import . "github.com/onsi/ginkgo/v2"
+	import . "github.com/onsi/ginkgo/v2"
 
 however some parts of the DSL may conflict with existing symbols in the user's code.
 
@@ -18,6 +18,7 @@ import (
 
 type Offset = ginkgo.Offset
 type FlakeAttempts = ginkgo.FlakeAttempts
+type MustPassRepeatedly = ginkgo.MustPassRepeatedly
 type Labels = ginkgo.Labels
 type PollProgressAfter = ginkgo.PollProgressAfter
 type PollProgressInterval = ginkgo.PollProgressInterval

integration/_fixtures/decorations_fixture/flaky_repeated/flaky_repeated_fixture_suite_test.go

@@ -0,0 +1,33 @@
+package decorations_fixture_test
+
+import (
+	"fmt"
+	"testing"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestDecorationsFixture(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "DecorationsFixture Suite")
+}
+
+var countFlake = 0
+var countRepeat = 0
+
+var _ = Describe("some decorated specs", func() {
+	It("passes eventually", func() {
+		countFlake += 1
+		if countFlake < 3 {
+			Fail("fail")
+		}
+	}, FlakeAttempts(3))
+
+	It("fails eventually", func() {
+		countRepeat += 1
+		if countRepeat >= 3 {
+			Fail(fmt.Sprintf("failed on attempt #%d", countRepeat))
+		}
+	}, MustPassRepeatedly(3))
+})

integration/_fixtures/decorations_fixture/invalid_decorations_flakeattempts_mustpassrepeatedly/invalid_decorations_flakeattempts_mustpassrepeatedly_suite_test.go

@@ -0,0 +1,19 @@
+package invalid_decorations_flakeattempts_mustpassrepeatedly_test
+
+import (
+	"testing"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestInvalidDecorations(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "InvalidDecorations Suite - MustPassRepeatedly and FlakeAttempts")
+}
+
+var _ = Describe("invalid decorators: mustpassrepeatedly and flakeattempts", FlakeAttempts(3), MustPassRepeatedly(3), func() {
+	It("never runs", func() {
+
+	})
+})

integration/_fixtures/decorations_fixture/invalid_decorations_focused_pending/invalid_decorations_focused_pending_suite_test.go

@@ -0,0 +1,19 @@
+package invalid_decorations_focused_pending_test
+
+import (
+	"testing"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestInvalidDecorations(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "InvalidDecorations Suite - Focused and Pending")
+}
+
+var _ = Describe("invalid decorators: focused and pending", Focus, Pending, func() {
+	It("never runs", func() {
+
+	})
+})

integration/_fixtures/decorations_fixture/decorations_fixture_suite_test.go → integration/_fixtures/decorations_fixture/offset_focus_pending/offset_focus_pending_fixture_suite_test.go

@@ -12,9 +12,10 @@ func TestDecorationsFixture(t *testing.T) {
 	RunSpecs(t, "DecorationsFixture Suite")
 }
 
-var count = 0
+var countFlake = 0
+var countRepeat = 0
 
-var _ = Describe("some decorated tests", func() {
+var _ = Describe("some decorated specs", func() {
 	Describe("focused", Focus, func() {
 		OffsetIt()
 	})
@@ -23,13 +24,6 @@ var _ = Describe("some decorated tests", func() {
 
 	})
 
-	It("passes eventually", func() {
-		count += 1
-		if count < 3 {
-			Fail("fail")
-		}
-	}, FlakeAttempts(3))
-
 	It("focused it", Focus, func() {
 		Ω(true).Should(BeTrue())
 	})

integration/decorations_test.go

@@ -13,23 +13,42 @@ var _ = Describe("Decorations", func() {
 		fm.MountFixture("decorations")
 	})
 
-	It("processes the various decorations", func() {
-		session := startGinkgo(fm.PathTo("decorations"), "-vv", "--no-color")
+	It("processes the Offset, Focus and Pending decorations", func() {
+		session := startGinkgo(fm.PathTo("decorations", "offset_focus_pending"), "-vv", "--no-color")
 		Eventually(session).Should(gexec.Exit(types.GINKGO_FOCUS_EXIT_CODE))
 
 		out := string(session.Out.Contents())
 		Ω(out).Should(MatchRegexp(
 			`P \[PENDING\]
-some decorated tests
-.*decorations_fixture_suite_test.go:\d+
+some decorated specs
+.*offset_focus_pending_fixture_suite_test.go:\d+
   pending it`,
 		))
+
 		Ω(out).ShouldNot(ContainSubstring("never_see_this_file"))
 	})
 
-	It("exits with a clear error if decorations are misconfigured", func() {
-		session := startGinkgo(fm.PathTo("decorations", "invalid_decorations"), "-v", "--no-color")
+	It("processes the FlakeAttempts and the MustPassRepeatedly decorations", func() {
+		session := startGinkgo(fm.PathTo("decorations", "flaky_repeated"), "-vv", "--no-color")
+		Eventually(session).Should(gexec.Exit(1))
+
+		Ω(session).Should(gbytes.Say("Ginkgo: Attempt #1 Failed.  Retrying..."))
+		Ω(session).Should(gbytes.Say("Ginkgo: Attempt #2 Failed.  Retrying..."))
+
+		Ω(session).Should(gbytes.Say("Ginkgo: Attempt #1 Passed.  Repeating..."))
+		Ω(session).Should(gbytes.Say("Ginkgo: Attempt #2 Passed.  Repeating..."))
+		Ω(session).Should(gbytes.Say("failed on attempt #3"))
+	})
+
+	It("exits with a clear error if decorations are misconfigured - focus and pending error", func() {
+		session := startGinkgo(fm.PathTo("decorations", "invalid_decorations_focused_pending"), "-v", "--no-color")
 		Eventually(session).Should(gexec.Exit(1))
 		Ω(session).Should(gbytes.Say("Invalid Combination of Decorators: Focused and Pending"))
 	})
+
+	It("exits with a clear error if decorations are misconfigured - flakeattempts and mustpassrepeatedly error", func() {
+		session := startGinkgo(fm.PathTo("decorations", "invalid_decorations_flakeattempts_mustpassrepeatedly"), "-v", "--no-color")
+		Eventually(session).Should(gexec.Exit(1))
+		Ω(session).Should(gbytes.Say("Invalid Combination of Decorators: FlakeAttempts and MustPassRepeatedly"))
+	})
 })

internal/group.go

@@ -118,6 +118,8 @@ func (g *group) initialReportForSpec(spec Spec) types.SpecReport {
 		ParallelProcess:             g.suite.config.ParallelProcess,
 		IsSerial:                    spec.Nodes.HasNodeMarkedSerial(),
 		IsInOrderedContainer:        !spec.Nodes.FirstNodeMarkedOrdered().IsZero(),
+		MaxFlakeAttempts:            spec.Nodes.GetMaxFlakeAttempts(),
+		MaxMustPassRepeatedly:       spec.Nodes.GetMaxMustPassRepeatedly(),
 	}
 }
 
@@ -299,16 +301,29 @@ func (g *group) run(specs Specs) {
 
 		g.suite.currentSpecReport.StartTime = time.Now()
 		if !skip {
-			maxAttempts := max(1, spec.FlakeAttempts())
-			if g.suite.config.FlakeAttempts > 0 {
+
+			var maxAttempts = 1
+
+			if g.suite.currentSpecReport.MaxMustPassRepeatedly > 0 {
+				maxAttempts = max(1, spec.MustPassRepeatedly())
+			} else if g.suite.config.FlakeAttempts > 0 {
 				maxAttempts = g.suite.config.FlakeAttempts
+				g.suite.currentSpecReport.MaxFlakeAttempts = maxAttempts
+			} else if g.suite.currentSpecReport.MaxFlakeAttempts > 0 {
+				maxAttempts = max(1, spec.FlakeAttempts())
 			}
+
 			for attempt := 0; attempt < maxAttempts; attempt++ {
 				g.suite.currentSpecReport.NumAttempts = attempt + 1
 				g.suite.writer.Truncate()
 				g.suite.outputInterceptor.StartInterceptingOutput()
 				if attempt > 0 {
-					fmt.Fprintf(g.suite.writer, "\nGinkgo: Attempt #%d Failed.  Retrying...\n", attempt)
+					if g.suite.currentSpecReport.MaxMustPassRepeatedly > 0 {
+						fmt.Fprintf(g.suite.writer, "\nGinkgo: Attempt #%d Passed.  Repeating...\n", attempt)
+					}
+					if g.suite.currentSpecReport.MaxFlakeAttempts > 0 {
+						fmt.Fprintf(g.suite.writer, "\nGinkgo: Attempt #%d Failed.  Retrying...\n", attempt)
+					}
 				}
 
 				g.attemptSpec(attempt == maxAttempts-1, spec)
@@ -318,8 +333,15 @@ func (g *group) run(specs Specs) {
 				g.suite.currentSpecReport.CapturedGinkgoWriterOutput += string(g.suite.writer.Bytes())
 				g.suite.currentSpecReport.CapturedStdOutErr += g.suite.outputInterceptor.StopInterceptingAndReturnOutput()
 
-				if g.suite.currentSpecReport.State.Is(types.SpecStatePassed | types.SpecStateSkipped | types.SpecStateAborted | types.SpecStateInterrupted) {
-					break
+				if g.suite.currentSpecReport.MaxMustPassRepeatedly > 0 {
+					if g.suite.currentSpecReport.State.Is(types.SpecStateFailureStates | types.SpecStateSkipped) {
+						break
+					}
+				}
+				if g.suite.currentSpecReport.MaxFlakeAttempts > 0 {
+					if g.suite.currentSpecReport.State.Is(types.SpecStatePassed | types.SpecStateSkipped | types.SpecStateAborted | types.SpecStateInterrupted) {
+						break
+					}
 				}
 			}
 		}