Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[skip-ci] TMT: initial enablement #820

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

[skip-ci] TMT: initial enablement #820

wants to merge 1 commit into from

Conversation

lsm5
Copy link
Member

@lsm5 lsm5 commented Sep 28, 2023 •
edited
Loading

For local runs:
tmt run --all will by default spawn virtual machines for running each test.

tmt run --all provision --how local will run tests on the local machine.

The test run will echo the results dir at the very beginning. The actual results can be found in:
$TEST_RESULTS_DIR/plans/$TEST_NAME/execute/data/guest/default-0/script-00-1/output.txt

so for example:
/var/tmp/tmt/run-022/plans/validate_test/execute/data/guest/default-0/script-00-1/output.txt

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 28, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lsm5

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@lsm5 lsm5 changed the title TMT: add validate test TMT: initial enablement Sep 28, 2023
@lsm5
Copy link
Member Author

lsm5 commented Sep 28, 2023

Feel free to ignore this PR until I ping you.

@packit-as-a-service
Copy link

Ephemeral COPR build failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT validate test failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT integration tests failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT unit tests failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT validate test failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT integration tests failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT validate test failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT unit tests failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT validate test failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT integration tests failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT unit tests failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT validate test failed. @containers/packit-build please check.

1 similar comment
@packit-as-a-service
Copy link

TMT validate test failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT unit tests failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT integration tests failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT validate test failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT unit tests failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT validate test failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT unit tests failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT integration tests failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT validate test failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT unit tests failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT integration tests failed. @containers/packit-build please check.

@packit-as-a-service
Copy link

TMT unit tests failed. @containers/packit-build please check.

@lsm5 lsm5 force-pushed the packit-tmt branch 2 times, most recently from 0724e19 to 6eebc2a Compare July 22, 2024 12:32
@lsm5
Copy link
Member Author

lsm5 commented Jul 22, 2024

@Luap99 I'm currently seeing these 3 failures in TMT integration tests:

not ok 167 plugin - test error message
# (from function `assert' in file test/helpers.bash, line 348,
#  in test file test/500-plugin.bats, line 43)
#   `assert '{"error":"plugin \"error-plugin\" failed: exit code 1, message: setup error"}'' failed
#  nsenter -n -m -w -t 23547 ip link set lo up
#  nsenter -n -m -w -t 23547 /usr/libexec/podman/netavark --rootless false --config /tmp/netavark_bats.FSq3wt/config --plugin-directory ./targets/release/examples/ setup /proc/23549/ns/net
# {"error":"unknown network driver \"error-plugin\""}
# [ rc=1 (expected) ]
# #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
# #|     FAIL: [no test name given]
# #| expected: '{"error":"plugin \"error-plugin\" failed: exit code 1, message: setup error"}'
# #|   actual: '{"error":"unknown network driver \"error-plugin\""}'
# #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
not ok 168 plugin - host-device
# (from function `die' in file test/helpers.bash, line 249,
#  from function `run_helper' in file test/helpers.bash, line 233,
#  from function `run_in_host_netns' in file test/helpers.bash, line 166,
#  from function `run_netavark' in file test/helpers.bash, line 136,
#  from function `run_netavark_plugins' in file test/500-plugin.bats, line 36,
#  in test file test/500-plugin.bats, line 54)
#   `run_netavark_plugins setup $(get_container_netns_path) <<<"$config"' failed
#  nsenter -n -m -w -t 23574 ip link set lo up
#  nsenter -n -m -w -t 23574 ip link add dummy0 type dummy
#  nsenter -n -m -w -t 23574 /usr/libexec/podman/netavark --rootless false --config /tmp/netavark_bats.yDYDEN/config --plugin-directory ./targets/release/examples/ setup /proc/23576/ns/net
# {"error":"unknown network driver \"host-device-plugin\""}
# [ rc=1 (** EXPECTED 0 **) ]
# #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
# #| FAIL: exit code is 1; expected 0
# #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
not ok 169 plugin - stderr
# (from function `die' in file test/helpers.bash, line 249,
#  from function `run_helper' in file test/helpers.bash, line 233,
#  from function `run_in_host_netns' in file test/helpers.bash, line 166,
#  from function `run_netavark' in file test/helpers.bash, line 136,
#  from function `run_netavark_plugins' in file test/500-plugin.bats, line 36,
#  in test file test/500-plugin.bats, line 69)
#   `run_netavark_plugins setup $(get_container_netns_path) <<<"$config"' failed
#  nsenter -n -m -w -t 23604 ip link set lo up
#  nsenter -n -m -w -t 23604 /usr/libexec/podman/netavark --rootless false --config /tmp/netavark_bats.RJ6FV4/config --plugin-directory ./targets/release/examples/ setup /proc/23606/ns/net
# {"error":"unknown network driver \"stderr-plugin\""}
# [ rc=1 (** EXPECTED 0 **) ]
# #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
# #| FAIL: exit code is 1; expected 0
# #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

@Luap99
Copy link
Member

Luap99 commented Jul 23, 2024

@Luap99 I'm currently seeing these 3 failures in TMT integration tests:

not ok 167 plugin - test error message
# (from function `assert' in file test/helpers.bash, line 348,
#  in test file test/500-plugin.bats, line 43)
#   `assert '{"error":"plugin \"error-plugin\" failed: exit code 1, message: setup error"}'' failed
#  nsenter -n -m -w -t 23547 ip link set lo up
#  nsenter -n -m -w -t 23547 /usr/libexec/podman/netavark --rootless false --config /tmp/netavark_bats.FSq3wt/config --plugin-directory ./targets/release/examples/ setup /proc/23549/ns/net
# {"error":"unknown network driver \"error-plugin\""}
# [ rc=1 (expected) ]
# #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
# #|     FAIL: [no test name given]
# #| expected: '{"error":"plugin \"error-plugin\" failed: exit code 1, message: setup error"}'
# #|   actual: '{"error":"unknown network driver \"error-plugin\""}'
# #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
not ok 168 plugin - host-device
# (from function `die' in file test/helpers.bash, line 249,
#  from function `run_helper' in file test/helpers.bash, line 233,
#  from function `run_in_host_netns' in file test/helpers.bash, line 166,
#  from function `run_netavark' in file test/helpers.bash, line 136,
#  from function `run_netavark_plugins' in file test/500-plugin.bats, line 36,
#  in test file test/500-plugin.bats, line 54)
#   `run_netavark_plugins setup $(get_container_netns_path) <<<"$config"' failed
#  nsenter -n -m -w -t 23574 ip link set lo up
#  nsenter -n -m -w -t 23574 ip link add dummy0 type dummy
#  nsenter -n -m -w -t 23574 /usr/libexec/podman/netavark --rootless false --config /tmp/netavark_bats.yDYDEN/config --plugin-directory ./targets/release/examples/ setup /proc/23576/ns/net
# {"error":"unknown network driver \"host-device-plugin\""}
# [ rc=1 (** EXPECTED 0 **) ]
# #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
# #| FAIL: exit code is 1; expected 0
# #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
not ok 169 plugin - stderr
# (from function `die' in file test/helpers.bash, line 249,
#  from function `run_helper' in file test/helpers.bash, line 233,
#  from function `run_in_host_netns' in file test/helpers.bash, line 166,
#  from function `run_netavark' in file test/helpers.bash, line 136,
#  from function `run_netavark_plugins' in file test/500-plugin.bats, line 36,
#  in test file test/500-plugin.bats, line 69)
#   `run_netavark_plugins setup $(get_container_netns_path) <<<"$config"' failed
#  nsenter -n -m -w -t 23604 ip link set lo up
#  nsenter -n -m -w -t 23604 /usr/libexec/podman/netavark --rootless false --config /tmp/netavark_bats.RJ6FV4/config --plugin-directory ./targets/release/examples/ setup /proc/23606/ns/net
# {"error":"unknown network driver \"stderr-plugin\""}
# [ rc=1 (** EXPECTED 0 **) ]
# #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
# #| FAIL: exit code is 1; expected 0
# #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Because you have to build the example plugins for the tests to work. As these are just examples they are not be packaged because they are totally useless outside of this test (although I guess if we want to turn on distro level gating tests we need a package anyway).
Maybe create a new package netavark-test-plugins that includes them? And then we need to fix the tests to allow us to change the plugin path.

@lsm5 lsm5 force-pushed the packit-tmt branch 8 times, most recently from 31bc099 to 436631e Compare July 30, 2024 13:33
@lsm5 lsm5 force-pushed the packit-tmt branch 2 times, most recently from b02370d to 92a4f0b Compare October 31, 2024 14:32
Luap99
Luap99 reviewed Nov 1, 2024
View reviewed changes
Makefile Outdated
Comment on lines 125 to 132
# The TEST_PLUGINS envvar is used by bats to specify path to test-plugins
# directory
ifdef TEST_PLUGINS
$(info Skipping examples build as TEST_PLUGINS is set)
integration: $(CARGO_TARGET_DIR)
else
integration: $(CARGO_TARGET_DIR) examples
endif
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should just drop the examples dependency instead of making it more complicated. We do not depend on build either so it is not much different.

rpm/netavark.spec
Requires: %{name} = %{epoch}:%{version}-%{release}

%description tests
%{summary}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had this discussion recently about the podman RHEL rpm that this is not supported. So we should make sure if this ever gets top RHEL we do not ship it outside of the gating env if possible (i.e. follow whatever we decided to do with the podman rpm).

Outside of the bats test we do not support these examples either upstream. So maybe add the description here that these are test only binaries for gating and not recommended/supported for any other use.

@lsm5 lsm5 force-pushed the packit-tmt branch 3 times, most recently from 30b1e5b to 6e98139 Compare December 26, 2024 13:53
@lsm5
Copy link
Member Author

lsm5 commented Dec 26, 2024

cs10 blocked on bridge-utils absence: https://bugzilla.redhat.com/show_bug.cgi?id=2334212

@lsm5 lsm5 force-pushed the packit-tmt branch 6 times, most recently from 0de4667 to 0db8f31 Compare December 30, 2024 09:20
@lsm5
TMT: initial enablement
a588807 
This commit adds `validate`, `unit`, and `integration` tests using
for aarch64 and x86_64 arches on all active Fedora and CentOS Stream
versions.

Also includes podman revdep test contributed for c9s by
Yuhui Jiang <yujiang@redhat.com>.
Ref: https://gitlab.com/redhat/centos-stream/rpms/netavark/-/merge_requests/59

Signed-off-by: Lokesh Mandvekar <lsm5@fedoraproject.org>
@Luap99
Copy link
Member

Luap99 commented Jan 7, 2025

cs10 blocked on bridge-utils absence: https://bugzilla.redhat.com/show_bug.cgi?id=2334212

@lsm5 FYI I don't think we need that package here since 71d8a5a

So just drop the dep.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Luap99 Luap99 Luap99 left review comments

Assignees
No one assigned
Labels
approved do-not-merge/work-in-progress
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@lsm5 @Luap99