Incomplete documentation on stochastic tests in TCK

[jroper]

From the code:

  /**
   * Override and return {@code true} in order to skip executing tests marked as {@code Stochastic}.
   * Such tests MAY sometimes fail even though the impl
   */
  public boolean skipStochasticTests() {
    return false;
  }

Even though the impl what? Under what circumstances may they fail? I'm never seeing them fail locally, but I see them fail quite frequently in CI, is there anything that should be documented here about the TCK in order for me to determine if the failures are expected or not?

[ktoso]

Ouch, seems we've cut the comment some time during the development, thanks for noticing! I'll dig out the full comment in a bit, but it wasn't anything very precise.

The only stochastic test we have is stochastic_spec103_mustSignalOnMethodsSequentially, it tries to see if any of the subscriber on* methods is called concurrently. Can you point me to a failure and impl you're getting?

AFAIR if it fails your impl most likely has a problem, just that it does not manifest on your machine.
It's a stochastic test because if this test passes it does not prove correctness of "does not call concurrently", it may have just been lucky.

[viktorklang]

@jroper Ping

[viktorklang]

@jroper Annual ping ;)