Introduce junit-platform-suite-engine #2416

mpkorstanje · 2020-09-21T16:01:22Z

Overview

Implements a test engine that allows declarative execution of test suites using
the @Suite annotation.

Internally the Suite Engine uses the JUnit Platform Launcher. The engine works
adding the test descriptors of the discovered engines and tests as children to its
own test descriptor.

package org.junit.platform.suite;

import org.junit.platform.suite.api.SelectPackages;

@Suite
@SelectPackages("org.junit.suite.testcases")
class SelectPackageSuite {

}

Is equivalent to:

import org.junit.platform.engine.discovery.DiscoverySelectors;
import org.junit.platform.launcher.Launcher;
import org.junit.platform.launcher.LauncherDiscoveryRequest;
import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder;
import org.junit.platform.launcher.core.LauncherFactory;

public class Main {

    public static void main(String[] args) {
        Launcher launcher = LauncherFactory.create();
        LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
                .selectors(DiscoverySelectors.selectPackage("org.junit.suite.testcases"))
                .build();
        launcher.execute(request);
    }
}

Issue: #744

I hereby agree to the terms of the JUnit Contributor License Agreement.

Definition of Done

There are no TODOs left in the code
Method preconditions are checked and documented in the method's Javadoc
Coding conventions (e.g. for logging) have been followed
Change is covered by automated tests including corner cases, errors, and exception handling
Public API has Javadoc and @API annotations
Change is documented in the User Guide and Release Notes
All continuous integration builds pass

mpkorstanje · 2020-09-22T22:09:05Z

This implementation is not done yet. However before continuing it would be good to give this a review to ensure that this is fundamentally the right way to go.

Currently this PR features:

A declarative test suite that can be executed with Junit 5;
The suite supports existing selectors from junit-platform-suite-api;
and adds several new selectors to ensure parity with DiscoverySelectors.
Supports transparent translation between unique ids of the engines test descriptors and the launchers test plan.
Adds an junit-platform-suite aggregator module.

Implementation

The suite engine converts an annotated class into a discovery request. This request is executed and the resulting test plan is mapped to a tree of test descriptors. In essence the tree of test descriptors are a view on the test plan. So suppose the discovery requests produces this test plan:

JUnit Jupiter
|- TestA
||- method1
||- method2
|- TestB
||- method1
||- method2

Then that test plan is mapped to a tree of test descriptors like so:

SuiteEngine
|- ExampleSuite
||-JUnit Jupiter
|||- TestA
||||- method1
||||- method2
|||- TestB
||||- method1
||||- method2

The unique identifiers are remapped by pre-pending the unique identifier of the suite. So:

junit-jupiter/TestA/method1() ->  junit-suite/ExampleSuite/junit-jupiter/TestA/method1()

When resolving tests by unique id this is reversed.

Known limitations

Only top level classes annotated with @Suite are recognized.
No support for a @DynamicSuite.
No before or after suite hooks
No parallel execution
The selectors that were added to junit-platform-suite-api are not supported by @RunWith(JUnitPlatform.class)
The unique id discovery selector has intentionally not been implemented as an annotation. The unique id representation does not seem suitable for this.
Likewise the method discovery selector has been omitted. I could not work out a usable annotation.

I don't think these limitations are insurmountable, however I would consider solving them out of scope for this PR.

codecov · 2020-09-23T22:25:21Z

Codecov Report

Merging #2416 (b9911f6) into main (e170011) will increase coverage by 0.02%.
The diff coverage is 94.32%.

@@             Coverage Diff              @@
##               main    #2416      +/-   ##
============================================
+ Coverage     90.55%   90.57%   +0.02%     
- Complexity     4718     4791      +73     
============================================
  Files           413      423      +10     
  Lines         11665    11865     +200     
  Branches        921      922       +1     
============================================
+ Hits          10563    10747     +184     
- Misses          825      834       +9     
- Partials        277      284       +7

Impacted Files	Coverage Δ	Complexity Δ
...orm/launcher/core/EngineExecutionOrchestrator.java	`100.00% <ø> (ø)`	`11.00 <0.00> (ø)`
...latform/launcher/core/LauncherDiscoveryResult.java	`100.00% <ø> (ø)`	`5.00 <0.00> (ø)`
...latform/suite/engine/IsPotentialTestContainer.java	`50.00% <50.00%> (ø)`	`4.00 <4.00> (?)`
...org/junit/platform/suite/engine/SuiteLauncher.java	`84.61% <84.61%> (ø)`	`5.00 <5.00> (?)`
...g/junit/platform/suite/engine/SuiteTestEngine.java	`87.50% <87.50%> (ø)`	`5.00 <5.00> (?)`
...nit/platform/suite/engine/SuiteTestDescriptor.java	`95.00% <95.00%> (ø)`	`16.00 <16.00> (?)`
.../commons/SuiteLauncherDiscoveryRequestBuilder.java	`96.15% <96.15%> (ø)`	`28.00 <28.00> (?)`
...t/platform/suite/engine/ClassSelectorResolver.java	`97.05% <97.05%> (ø)`	`15.00 <15.00> (?)`
...rm/suite/commons/AdditionalDiscoverySelectors.java	`97.43% <97.43%> (ø)`	`16.00 <16.00> (?)`
.../main/java/org/junit/platform/engine/UniqueId.java	`82.08% <100.00%> (+0.27%)`	`24.00 <1.00> (+1.00)`
... and 19 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e170011...b9911f6. Read the comment docs.

mpkorstanje · 2020-10-12T11:52:33Z

So @marcphilipp what do you think? If there are general aspects you have doubts about -more a problem with the concept of a suite engine, rather then this particular implementation- I'd like to hear that too.

I did create a small demo to play around with:

/~https://github.com/mpkorstanje/junit-platform-suite-demo

kriegfrj · 2020-10-15T11:33:37Z

So @marcphilipp what do you think? If there are general aspects you have doubts about -more a problem with the concept of a suite engine, rather then this particular implementation- I'd like to hear that too.

I did create a small demo to play around with:

/~https://github.com/mpkorstanje/junit-platform-suite-demo

For reference, this technique of a "meta TestEngine" (that recursively calls other engines using the launcher API) was also a feature of the BundleEngine that we implemented as part of Bnd for testing in OSGi. The BundleEngine takes care of all the classloading magic that needs to happen in an OSGi environment and passes the loaded classes off to the other engines to run the actual tests. See /~https://github.com/bndtools/bnd/blob/master/biz.aQute.tester.junit-platform/src/aQute/tester/bundle/engine/BundleEngine.java

mpkorstanje · 2020-10-15T22:00:18Z

Good to know that the idea has already been tested! Any practical problems you've ran into with regards to IDEs and other bits of tooling?

this technique of a "meta TestEngine" (that recursively calls other engines using the launcher API) was also a feature of the BundleEngine

Unless I'm misunderstanding the purpose and function of findEngines it would appear that the Bundle engine is neither recursive nor using the Launcher API.

The line .filter(engine -> engine.getId() != BundleEngine.ENGINE_ID) would prevent the engine from calling itself. And the engines are loaded directly via the ServiceLoader rather then org.junit.platform.launcher.Launcher.

kriegfrj · 2020-10-15T22:55:02Z

Good to know that the idea has already been tested! Any practical problems you've ran into with regards to IDEs and other bits of tooling?

There are bits of the code in various places that softly assume that the Engine will be at the top of the hierarchy. For example, the naming convention for the segments of the UniqueId for the tests. In our hierarchy, the BundleEngine's segment uses "engine:bnd-bundle-engine" and the sub-engines have "sub-engine:junit-jupiter", etc. But these assumptions seem to be "soft" and can be worked around. We've been using the BundleEngine in anger extensively in the osgi-test project (/~https://github.com/osgi/osgi-test) - it adds testdescriptor nodes representing the bundle (and fragment if there is one) at the top of the hierarchy.

this technique of a "meta TestEngine" (that recursively calls other engines using the launcher API) was also a feature of the BundleEngine

Unless I'm misunderstanding the purpose and function of findEngines it would appear that the Bundle engine is neither recursive nor using the Launcher API.

The line .filter(engine -> engine.getId() != BundleEngine.ENGINE_ID) would prevent the engine from calling itself. And the engines are loaded directly via the ServiceLoader rather then org.junit.platform.launcher.Launcher.

On the contrary, it seems you've understood the purpose and function well. 😄

Sorry, I used the terms "recursive" and "launcher API" a bit too loosely. By "recursive" I meant one TestEngine invoking another, which as you point out is not strictly recursive - we explicitly prevent recursion as you noted. And no, we don't use the launcher API - we bypass it to invoke the engines directly.

marcphilipp · 2020-10-25T16:20:00Z

@mpkorstanje Sorry for the delay! This is still on my list, just haven't had the time to do a proper review, yet.

jlink · 2020-11-09T06:49:25Z

I haven’t checked the implementation so maybe this has already been discussed: How are the engines called from this one being prevented from running their tests? If they are not the tests in the suite will be run twice, right?

mpkorstanje · 2020-11-09T12:35:05Z

How are the engines called from this one being prevented from running their tests

They are not. Currently to avoid this the suite engine and all other engines that are not part of the suite should be explicitly excluded in the discovery request. While perhaps unexpected I don't think we can implement a less surprising default behavior. Did you have anything in mind?

Edit: included -> excluded

jlink · 2020-11-09T13:46:12Z

Currently to avoid this the suite engine and all other engines that are not part of the suite should be explicitly included in the discovery request. While perhaps unexpected I don't think we can implement a less surprising default behavior. Did you have anything in mind?

I think the problem occurs when you run all tests (from IDE or Gradle or Maven) since then we'd see some tests being executed twice. Maybe a reasonable default behaviour would be to NOT execute a suite at all when the request includes other test engines?

mpkorstanje · 2020-11-09T15:57:49Z

I'm not sure if that would be reasonable.

It is hard to understand why a test is not picked up -- nothing happened. On the other hand when a test is executed twice it will be apparent when looking at IDEAs and JUnits tree output that both the test engine and the suite engine ran the test. This makes the solution easier to find because you know what to stop from happening.

And because both the problem of not having the suite executed and the problem of having tests executed twice result in the same solution it would be preferable to make the problem observable.
I don't believe IDEA or Maven explicitly include any engine filters by default. They're using either the class or package selectors. If a package or class contains tests for different engines I would expect all to be executed, regardless of whether engine executes the same set of tests or a completely different set of tests.

Tests being executed twice won't happen in normal circumstances. Users will have to explicitly include a dependency on the suite engine. So it shouldn't surprise any one who's been used to regular junit.
The same problem currently exists when using using JUnit 5 Suites in Junit 4.

dosomder · 2020-11-11T11:22:45Z

@mpkorstanje Great work, looking forward to this so I can migrate to cucumber-junit-platform-engine

...platform-suite-engine/src/main/java/org/junit/platform/suite/engine/SuiteTestDescriptor.java

carlspring · 2020-12-21T15:21:32Z

Hi,

Are there any updates on this? What still needs to be done/tested? Is there any ETA on when this could be accepted, merged and released?

Kind regards,

Martin

marcphilipp · 2020-12-21T15:26:50Z

@carlspring Still on my list and not forgotten, just a little short on time these days. Will try to get it in for 5.8 M1, can't promise any date, though.

carlspring · 2020-12-21T15:28:42Z

Hi @marcphilipp ,

Thanks for your quick response!

Would you mind clarifying what still needs to be done and how much of an effort it would be? Also, just off the record and quite optimistically, when do you suppose 5.8-M1 would come out (if this feature does make it in)? Would this add complete support for test suites, or would more work be required?

Thanks again! :)

mpkorstanje · 2020-12-27T20:01:23Z

@carlspring

Would you mind clarifying what still needs to be done and how much of an effort it would be? ... Would this add complete support for test suites, or would more work be required?

I think the first two posts of this MR should answer that. If that leaves you with any questions you can build this locally and try it out.

If you are looking to contribute the User Guide and Release notes have not been updated yet. I was waiting for Marcs review before writing these but in the mean time I'd welcome a PR too. Please also feel free to give the implementation a review.

@suite

Implements a test engine that allows declarative execution of test suites using the `@Suite` annotation. Internally the Suite Engine uses the JUnit Platform Launcher. The engine works by mapping the `TestIdentifier` used by the launcher to `TestDescriptor` used by the engine during discovery and execution. ``` package org.junit.platform.suite; import org.junit.platform.suite.api.SelectPackages; @suite @SelectPackages("org.junit.suite.testcases") class SelectPackageSuite { } ``` Is equivalent to: ``` import org.junit.platform.engine.discovery.DiscoverySelectors; import org.junit.platform.launcher.Launcher; import org.junit.platform.launcher.LauncherDiscoveryRequest; import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder; import org.junit.platform.launcher.core.LauncherFactory; public class Main { public static void main(String[] args) { Launcher launcher = LauncherFactory.create(); LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request() .selectors(DiscoverySelectors.selectPackage("org.junit.suite.testcases")) .build(); launcher.execute(request); } } ``` The suite engine converts an annotated class into a discovery request. This request is executed and the resulting test plan is mapped to a tree of test descriptors. In essence the tree of test descriptors are a view on the test plan. So suppose the discovery requests produces this test plan: ``` JUnit Jupiter |- TestA ||- method1 ||- method2 |- TestB ||- method1 ||- method2 ``` Then that test plan is mapped to a tree of test descriptors like so: ``` SuiteEngine |- ExampleSuite ||-JUnit Jupiter |||- TestA ||||- method1 ||||- method2 |||- TestB ||||- method1 ||||- method2 ```` The unique identifiers are remapped by pre-pending the unique identifier of the suite. So: ``` junit-jupiter/TestA/method1() -> junit-suite/ExampleSuite/junit-jupiter/TestA/method1() ``` Issue: junit-team#744

…uite/engine/SuiteLauncher.java Co-authored-by: Marc Philipp <marc@gradle.com>

mpkorstanje · 2021-02-07T17:31:16Z

Very thorough work! I only found a few minor things. When this PR is merged, I'd like to release 5.8.0-M1 so we can get feedback from the community.

Cheers. Looking forward.

mpkorstanje · 2021-02-07T17:55:13Z

Btw. Should I squash all commits before they're merged?

...rm-suite-engine/src/main/java/org/junit/platform/suite/engine/DiscoverySelectorResolver.java

marcphilipp · 2021-02-07T18:49:29Z

Should I squash all commits before they're merged?

Don't bother, I'll do that when merging.

marcphilipp · 2021-02-09T13:34:07Z

Thanks so much for all the hard work, @mpkorstanje! 👍

I sent a tweet from the team account in your honor. https://twitter.com/junitteam/status/1359130553224749059

mpkorstanje · 2021-02-09T15:07:26Z

You're welcome. This was fun to do.

dosomder · 2021-02-10T09:11:00Z

@mpkorstanje thanks for your work. Now that this PR is merged, could you update the answer here? https://stackoverflow.com/questions/64587739/cucumberoptions-in-cucumber-junit-platform-engine/64598725#64598725

mpkorstanje · 2021-02-10T19:24:04Z

No, not any time soon, but feel free to post another answer.

@suite

Implements a test engine that allows declarative execution of test suites using the `@Suite` annotation. Internally the Suite Engine uses the JUnit Platform Launcher. The engine works adding the test descriptors of the discovered engines and tests as children to its own test descriptor. ```java package org.junit.platform.suite; import org.junit.platform.suite.api.SelectPackages; @suite @SelectPackages("org.junit.suite.testcases") class SelectPackageSuite { } ``` Is equivalent to: ```java import org.junit.platform.engine.discovery.DiscoverySelectors; import org.junit.platform.launcher.Launcher; import org.junit.platform.launcher.LauncherDiscoveryRequest; import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder; import org.junit.platform.launcher.core.LauncherFactory; public class Main { public static void main(String[] args) { Launcher launcher = LauncherFactory.create(); LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request() .selectors(DiscoverySelectors.selectPackage("org.junit.suite.testcases")) .build(); launcher.execute(request); } } ``` Resolves junit-team#744.

mpkorstanje marked this pull request as ready for review September 22, 2020 22:09

mpkorstanje force-pushed the junit-platform-suite-engine branch 2 times, most recently from d2c8982 to 04e7397 Compare September 23, 2020 22:17

mpkorstanje force-pushed the junit-platform-suite-engine branch from 45c13bb to 3abe7c4 Compare September 23, 2020 22:42

marcphilipp self-requested a review September 25, 2020 10:25

This was referenced Oct 6, 2020

change doc for junit5 cucumber/docs#487

Open

Introduce support for declarative test suites #744

Closed

sbrannen added component: Platform theme: suites type: new feature labels Nov 4, 2020

mpkorstanje mentioned this pull request Dec 18, 2020

Introduce file-based declarative test suite support #1243

Closed

mpkorstanje commented Dec 21, 2020

View reviewed changes

...platform-suite-engine/src/main/java/org/junit/platform/suite/engine/SuiteTestDescriptor.java Outdated Show resolved Hide resolved

carlspring mentioned this pull request Dec 21, 2020

Investigate if and how we can extract the integration tests into suites carlspring/s3fs-nio#140

Closed

3 tasks

mpkorstanje and others added 2 commits February 7, 2021 17:27

Update junit-platform-suite-engine/src/main/java/org/junit/platform/s…

da654f2

…uite/engine/SuiteLauncher.java Co-authored-by: Marc Philipp <marc@gradle.com>

Rename Configuration to ConfigurationParameter

4ade1c4

mpkorstanje force-pushed the junit-platform-suite-engine branch from 8fe44a2 to 4ade1c4 Compare February 7, 2021 16:30

mpkorstanje added 4 commits February 7, 2021 17:42

Only discover test in suit for engines direct descendents

94e47a2

Delay prune until after suits have been discovered

a7bb955

Add note about aggregator

6b4a1f3

Use note styling

b9911f6

marcphilipp reviewed Feb 7, 2021

View reviewed changes

...rm-suite-engine/src/main/java/org/junit/platform/suite/engine/DiscoverySelectorResolver.java Outdated Show resolved Hide resolved

mpkorstanje mentioned this pull request Feb 9, 2021

Cucumber JUnit 5 Change of behaviour: Cannot have multiple spring boot cucumber contexts cucumber/cucumber-jvm#2230

Closed

marcphilipp approved these changes Feb 9, 2021

View reviewed changes

marcphilipp changed the title ~~Implement junit-platform-suite-engine~~ Introduce junit-platform-suite-engine Feb 9, 2021

marcphilipp merged commit b2a9810 into junit-team:main Feb 9, 2021

mpkorstanje deleted the junit-platform-suite-engine branch February 9, 2021 13:17

This was referenced Mar 2, 2021

Option cucumber.features does not work in cucumber.properties file cucumber/cucumber-jvm#2245

Closed

Custom predicate cucumber/cucumber-jvm#2258

Closed

jkschneider mentioned this pull request Mar 15, 2021

JUnit 4 ClassPathSuite runner recipe openrewrite/rewrite-testing-frameworks#66

Closed

mpkorstanje mentioned this pull request Apr 14, 2021

Discover tests in suites packaged into jars, e.g. TCKs #2594

Closed

2 tasks

mpkorstanje mentioned this pull request May 2, 2021

How can I work with multiple spring boot cucumber contexts? cucumber/cucumber-jvm#2294

Closed

This was referenced May 10, 2021

Use the JUnit 5 @Suite. charleech/arquillian-junit5-payara#10

Closed

Using with JUnit 5 @RunWith(JUnitPlatform.class) causes multiple deployment. ingwarsw/arquillian-suite-extension#71

Closed

mpkorstanje mentioned this pull request Jun 6, 2021

Parallel execution with surefire and failsafe doesn't work with junit-vintage-engine cucumber/docs#627

Open

mpkorstanje mentioned this pull request Sep 17, 2021

JUnit 5: Allow resource-based testing gradle/gradle#4773

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce junit-platform-suite-engine #2416

Introduce junit-platform-suite-engine #2416

mpkorstanje commented Sep 21, 2020 •

edited

Loading

mpkorstanje commented Sep 22, 2020 •

edited

Loading

codecov bot commented Sep 23, 2020 •

edited

Loading

mpkorstanje commented Oct 12, 2020 •

edited

Loading

kriegfrj commented Oct 15, 2020

mpkorstanje commented Oct 15, 2020

kriegfrj commented Oct 15, 2020 •

edited

Loading

marcphilipp commented Oct 25, 2020

jlink commented Nov 9, 2020

mpkorstanje commented Nov 9, 2020 •

edited

Loading

jlink commented Nov 9, 2020

mpkorstanje commented Nov 9, 2020 •

edited

Loading

dosomder commented Nov 11, 2020 •

edited

Loading

carlspring commented Dec 21, 2020

marcphilipp commented Dec 21, 2020

carlspring commented Dec 21, 2020 •

edited

Loading

mpkorstanje commented Dec 27, 2020

mpkorstanje commented Feb 7, 2021

mpkorstanje commented Feb 7, 2021

marcphilipp commented Feb 7, 2021

marcphilipp commented Feb 9, 2021

mpkorstanje commented Feb 9, 2021

dosomder commented Feb 10, 2021

mpkorstanje commented Feb 10, 2021

Introduce junit-platform-suite-engine #2416

Introduce junit-platform-suite-engine #2416

Conversation

mpkorstanje commented Sep 21, 2020 • edited Loading

Overview

Definition of Done

mpkorstanje commented Sep 22, 2020 • edited Loading

Implementation

Known limitations

codecov bot commented Sep 23, 2020 • edited Loading

Codecov Report

mpkorstanje commented Oct 12, 2020 • edited Loading

kriegfrj commented Oct 15, 2020

mpkorstanje commented Oct 15, 2020

kriegfrj commented Oct 15, 2020 • edited Loading

marcphilipp commented Oct 25, 2020

jlink commented Nov 9, 2020

mpkorstanje commented Nov 9, 2020 • edited Loading

jlink commented Nov 9, 2020

mpkorstanje commented Nov 9, 2020 • edited Loading

dosomder commented Nov 11, 2020 • edited Loading

carlspring commented Dec 21, 2020

marcphilipp commented Dec 21, 2020

carlspring commented Dec 21, 2020 • edited Loading

mpkorstanje commented Dec 27, 2020

mpkorstanje commented Feb 7, 2021

mpkorstanje commented Feb 7, 2021

marcphilipp commented Feb 7, 2021

marcphilipp commented Feb 9, 2021

mpkorstanje commented Feb 9, 2021

dosomder commented Feb 10, 2021

mpkorstanje commented Feb 10, 2021

mpkorstanje commented Sep 21, 2020 •

edited

Loading

mpkorstanje commented Sep 22, 2020 •

edited

Loading

codecov bot commented Sep 23, 2020 •

edited

Loading

mpkorstanje commented Oct 12, 2020 •

edited

Loading

kriegfrj commented Oct 15, 2020 •

edited

Loading

mpkorstanje commented Nov 9, 2020 •

edited

Loading

mpkorstanje commented Nov 9, 2020 •

edited

Loading

dosomder commented Nov 11, 2020 •

edited

Loading

carlspring commented Dec 21, 2020 •

edited

Loading