[QE] Compactify e2e tests #3167

jsliacan · 2022-05-09T08:25:58Z

This PR contains the following changes

include the relevant clicumber code in this codebase, as suggested in [e2e] integrate clicumber as part of extended test #2538 and [e2e] integrate clicumber as part of extended test #3210
update dependencies (cucumber/godog/...)
compactify features/scenarios to fit the cucumber paradigms
config.feature retains its old form for now (should be replaced by an integration test of similar extent [QE] Move config test from e2e to integration #3249)

adrianriobo · 2022-07-06T10:42:14Z

Does it make sense keep the clicumber integrated code under e2e?
Or may keep e2e only with suites and features and move the clicumber code to extended?

WDYT?

jsliacan · 2022-07-06T11:08:32Z

@adrianriobo , I moved it directly into testsuite because testsuite was "extending" the clicumber suite. That said, I am happy to put it wherever you think is better, I have no real opinions either way.

EDIT: It's in extended now as you suggested.

jsliacan · 2022-07-15T11:08:48Z

/retest

anjannath · 2022-08-01T11:57:40Z

/test

openshift-ci · 2022-08-01T11:57:43Z

@anjannath: The /test command needs one or more targets.
The following commands are available to trigger required jobs:

/test e2e-crc
/test images
/test integration-crc

Use /test all to run all jobs.

In response to this:

/test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jsliacan · 2022-09-15T11:28:13Z

/retest

anjannath · 2022-09-28T08:40:39Z

@jsliacan pr needs rebase

anjannath · 2022-09-28T14:18:54Z

nice speedup for e2e, integration tests!

test time	e2e	integration
this pr	1h52m46s	1h12m18s
in main	2h31m8s	1h23m38s

anjannath · 2022-09-28T14:47:35Z

Include clicumber code, update dependencies, compactify feature files

i guess these three things are tightly coupled and hard to split into individual commits without rewriting the feature files multiple times for each commit.. better to also include the information in the pr description as commit log

anjannath · 2022-09-29T05:47:33Z

leave out config.feature (should be replaced by an integration test of similar extent #3249)

@jsliacan maybe keep the config.feature for now (if it works with the current changes) and remove it when we have something similar in the integration tests? i.e when working on fixing #3249

praveenkumar · 2022-09-30T04:47:22Z

developing.adoc

-End-to-end (e2e) tests use the link:http://github.com/code-ready/clicumber[Clicumber] package to provide basic functionality for testing CLI binaries.
-
-Clicumber allows running commands in a persistent shell instance (`bash`, `tcsh`, `zsh`, Command Prompt, or PowerShell), assert its outputs (standard output, standard error, or exit code), check configuration files, and so on. The general functionality of Clicumber is then extended by CodeReady Containers specific test code to cover the whole functionality of CodeReady Containers.
+End-to-end (e2e) tests borrow code from link:http://github.com/code-ready/clicumber[Clicumber] package to provide basic functionality for testing CLI binaries. This facilitates running commands in a persistent shell instance (`bash`, `tcsh`, `zsh`, Command Prompt, or PowerShell), assert its outputs (standard output, standard error, or exit code), check configuration files, and so on. The general functionality of Clicumber is then extended by CodeReady Containers specific test code to cover the whole functionality of CodeReady Containers.


Shouldn't we update /~https://github.com/code-ready/clicumber codebase to use latest version and then consume it as we are doing now?

For easier maintenance etc., the idea was to include it on our side instead. I mention #3210 in the description above to hint at this.

@jsliacan which mean we are not using /~https://github.com/code-ready/clicumber anymore?

Correct, after this PR we won't be importing clicumber project.

anjannath · 2022-09-30T08:41:44Z

better to exclude the config.feature changes from the first commit itself than removing it and re-adding

openshift-ci · 2022-09-30T11:06:13Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: anjannath

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [anjannath]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

praveenkumar · 2022-10-06T08:28:08Z

test/e2e/features/basic.feature

        When executing "crc setup --check-only" fails
        Then executing crc start command fails
        And executing single crc setup command succeeds

-    @linux


Why we are removing this test?

I removed it because it cannot run in parallel with the rest of the scenarios in the Feature. Tagging would make it hard to read & cluttered if I included it in the next Scenario. So it's going to go in the integration test. I'll list it in the issue of things to move to integration.

praveenkumar · 2022-10-06T08:30:02Z

test/e2e/features/basic.feature

+        # Request start with monitoring stack
+        * setting config property "enable-cluster-monitoring" to value "true" succeeds
+        * setting config property "memory" to value "16000" succeeds
+        Given executing crc setup command succeeds


Also now we are only testing if user provide correct mem value what about the error it display if user doesn't provide the correct mem like it use to be?

there's a separate config.feature and its covered there, so should be good i think

@anjannath I don't see any config.feature file, I think @jsliacan mentioned he will create separate task issue for those missing stuff. so once that in place we should take this PR in and iterate over.

config.feature exists from before, this was removed in an earlier iteration of the PR but the current PR branch doesn't remove that or am i missing something?

Yes, I'll make an issue and it'll go into the config integration test. I don't think it's in the current config.feature. It's not the most important testcase though and the minimum we declare should work is 14GiB (docs), so maybe I'll use this value instead of 16000 that we had.

I updated the issue with both things @praveenkumar noticed: #3249 and altered the basic.feature and config.feature a little. I moved the "missing setup" scenario to config.feature (which will be replaced by an integration test) and cleaned up scenarios a little more in basic.feature which will stay in e2e.

praveenkumar · 2022-10-06T08:31:43Z

test/e2e/features/basic.feature

-
-    @darwin @linux @windows
-    Scenario: CRC status and disk space check
+        * stdout should match "(?s)(.*)oc login -u developer https:\/\/api\.crc\.testing:6443(.*)$"


what does * means because some place you used And and some place it is replaced by * ?

Gherkin also supports using an asterisk (*) in place of any of the normal step keywords. This can be helpful when you have some steps that are effectively a list of things, so you can express it more like bullet points where otherwise the natural language of And etc might not read so elegantly.

from: https://cucumber.io/docs/gherkin/reference/#Asterisk

Yes yes, it's just a bullet point. I use them somewhere when it's an obvious "enumeration" of things to check. But it's just for fun in a way, And would've worked just as well.

- include relevant clicumber code in this codebase, as suggested in crc-org#2538 and crc-org#3210 - update dependencies (cucumber/godog/...) - compactify features/scenarios to fit the cucumber paradigms

openshift-ci · 2022-10-10T14:47:43Z

New changes are detected. LGTM label has been removed.

openshift-ci bot requested review from gbraad and praveenkumar May 9, 2022 08:26

jsliacan added do-not-merge/hold do-not-merge/work-in-progress labels May 9, 2022

openshift-ci bot added the needs-rebase label Jun 1, 2022

jsliacan force-pushed the e2e-speedup branch from 168e073 to 1504da2 Compare June 15, 2022 09:33

openshift-ci bot removed the needs-rebase label Jun 15, 2022

openshift-ci bot added the needs-rebase label Jun 25, 2022

jsliacan force-pushed the e2e-speedup branch from 1504da2 to 46f8c47 Compare July 6, 2022 09:14

openshift-ci bot removed the needs-rebase label Jul 6, 2022

jsliacan changed the title ~~[Hold] E2e: reducing number of start/stop occasions~~ [Hold] E2e: compactify e2e tests Jul 6, 2022

openshift-ci bot removed the do-not-merge/work-in-progress label Jul 6, 2022

jsliacan changed the title ~~[Hold] E2e: compactify e2e tests~~ E2e: compactify e2e tests Jul 6, 2022

jsliacan changed the title ~~E2e: compactify e2e tests~~ [QE] Compactify e2e tests Jul 6, 2022

jsliacan force-pushed the e2e-speedup branch 5 times, most recently from bd7da7f to 16b5ffc Compare July 15, 2022 10:45

jsliacan force-pushed the e2e-speedup branch 5 times, most recently from 1baf189 to a0cd8c3 Compare July 26, 2022 11:28

anjannath removed the do-not-merge/hold label Jul 27, 2022

openshift-merge-robot added the needs-rebase label Sep 23, 2022

jsliacan force-pushed the e2e-speedup branch from f98aa42 to fa373dd Compare September 28, 2022 09:02

openshift-merge-robot removed the needs-rebase label Sep 28, 2022

anjannath approved these changes Sep 28, 2022

View reviewed changes

openshift-ci bot assigned anjannath Sep 28, 2022

openshift-ci bot added lgtm approved labels Sep 28, 2022

praveenkumar reviewed Sep 30, 2022

View reviewed changes

jsliacan force-pushed the e2e-speedup branch from fa373dd to b9cf606 Compare September 30, 2022 07:52

openshift-ci bot removed the lgtm label Sep 30, 2022

jsliacan force-pushed the e2e-speedup branch 4 times, most recently from 39ea2ac to c820548 Compare September 30, 2022 09:16

anjannath approved these changes Sep 30, 2022

View reviewed changes

openshift-ci bot added the lgtm label Sep 30, 2022

anjannath mentioned this pull request Sep 30, 2022

Move away from the code-ready namespace #3264

Closed

praveenkumar reviewed Oct 6, 2022

View reviewed changes

Include clicumber code, update dependencies, compactify feature files

74aa719

- include relevant clicumber code in this codebase, as suggested in crc-org#2538 and crc-org#3210 - update dependencies (cucumber/godog/...) - compactify features/scenarios to fit the cucumber paradigms

jsliacan force-pushed the e2e-speedup branch from c820548 to 74aa719 Compare October 10, 2022 14:47

openshift-ci bot removed the lgtm label Oct 10, 2022

praveenkumar merged commit 82ca273 into crc-org:main Oct 11, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[QE] Compactify e2e tests #3167

[QE] Compactify e2e tests #3167

jsliacan commented May 9, 2022 •

edited

Loading

adrianriobo commented Jul 6, 2022

jsliacan commented Jul 6, 2022 •

edited

Loading

jsliacan commented Jul 15, 2022

anjannath commented Aug 1, 2022

openshift-ci bot commented Aug 1, 2022

jsliacan commented Sep 15, 2022

anjannath commented Sep 28, 2022

anjannath commented Sep 28, 2022 •

edited

Loading

anjannath commented Sep 28, 2022 •

edited

Loading

anjannath commented Sep 29, 2022 •

edited

Loading

praveenkumar Sep 30, 2022

jsliacan Sep 30, 2022

praveenkumar Oct 6, 2022

jsliacan Oct 6, 2022

anjannath commented Sep 30, 2022

openshift-ci bot commented Sep 30, 2022

praveenkumar Oct 6, 2022

jsliacan Oct 10, 2022

praveenkumar Oct 6, 2022

anjannath Oct 10, 2022

praveenkumar Oct 10, 2022

anjannath Oct 10, 2022

jsliacan Oct 10, 2022

jsliacan Oct 10, 2022

praveenkumar Oct 6, 2022

anjannath Oct 10, 2022

jsliacan Oct 10, 2022

openshift-ci bot commented Oct 10, 2022

[QE] Compactify e2e tests #3167

[QE] Compactify e2e tests #3167

Conversation

jsliacan commented May 9, 2022 • edited Loading

adrianriobo commented Jul 6, 2022

jsliacan commented Jul 6, 2022 • edited Loading

jsliacan commented Jul 15, 2022

anjannath commented Aug 1, 2022

openshift-ci bot commented Aug 1, 2022

jsliacan commented Sep 15, 2022

anjannath commented Sep 28, 2022

anjannath commented Sep 28, 2022 • edited Loading

anjannath commented Sep 28, 2022 • edited Loading

anjannath commented Sep 29, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anjannath commented Sep 30, 2022

openshift-ci bot commented Sep 30, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-ci bot commented Oct 10, 2022

jsliacan commented May 9, 2022 •

edited

Loading

jsliacan commented Jul 6, 2022 •

edited

Loading

anjannath commented Sep 28, 2022 •

edited

Loading

anjannath commented Sep 28, 2022 •

edited

Loading

anjannath commented Sep 29, 2022 •

edited

Loading