Multiple test modules fail to run if invoked directly #90583

sobolevn · 2022-01-18T18:41:39Z

BPO	46425
Nosy	@terryjreedy, @jaraco, @asvetlov, @ethanfurman, @zware, @serhiy-storchaka, @corona10, @miss-islington, @sobolevn
PRs	bpo-46425: fix direct invokation of multiple test modules #30666 bpo-46425: fix direct invocation of `test_sqlite3` #30676 bpo-46425: fix direct invocation of `test_contextlib` #30681 bpo-46425: fix direct invocation of `test_importlib` #30682 [3.10] bpo-46425: Fix direct invocation of multiple test modules (GH-30666) #30699 [3.9] bpo-46425: Fix direct invocation of multiple test modules (GH-30666) #30700 bpo-46425: fix direct invocation of `asyncio` tests #30725 bpo-46425: fix direct invocation of `test_traceback` #30746 [3.10] bpo-46425: fix direct invocation of `asyncio` tests (GH-30725) #30782 [3.9] bpo-46425: fix direct invocation of `asyncio` tests (GH-30725) #30783 bpo-46425: fix direct invocation of `test_fileutils` and `test_zoneinfo` #30792 bpo-46425: Partially revert GH-30682 #30799 bpo-46425: revert changes to `importlib.metadata` and `importlib.resources` tests #30800 bpo-46474: sync with importlib_metadata 4.10.0 #30802 bpo-46474: Avoid REDoS in EntryPoint.pattern (sync with importlib_metadata 4.10.1) #30803

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-01-18.18:41:39.303>
labels = ['type-bug', 'tests', '3.9', '3.10', '3.11']
title = 'Multiple test modules fail to run if invoked directly'
updated_at = <Date 2022-01-23.02:38:46.011>
user = '/~https://github.com/sobolevn'

bugs.python.org fields:

activity = <Date 2022-01-23.02:38:46.011>
actor = 'jaraco'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2022-01-18.18:41:39.303>
creator = 'sobolevn'
dependencies = []
files = []
hgrepos = []
issue_num = 46425
keywords = ['patch']
message_count = 18.0
messages = ['410879', '410880', '410881', '410886', '410887', '410911', '410924', '410999', '411000', '411088', '411247', '411250', '411252', '411264', '411265', '411266', '411267', '411334']
nosy_count = 9.0
nosy_names = ['terry.reedy', 'jaraco', 'asvetlov', 'ethan.furman', 'zach.ware', 'serhiy.storchaka', 'corona10', 'miss-islington', 'sobolevn']
pr_nums = ['30666', '30676', '30681', '30682', '30699', '30700', '30725', '30746', '30782', '30783', '30792', '30799', '30800', '30802', '30803']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue46425'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

The text was updated successfully, but these errors were encountered:

sobolevn · 2022-01-18T18:41:39Z

While working on a similar issue for test_typing.py (https://bugs.python.org/issue46416) I wondered: how many other modules also have this problem?

So, after several hours of local testing, I got several problems:

Lib/test/test_compileall.py
Uses relative . import that cannot be resolved
Lib/test/test_distutils.py
Is missing import unittest to run unittest.main()
Lib/test/test_dtrace.py
Calls undefined test_main() instead of unittest.main()
Lib/test/test_tools/test_freeze.py
Uses relative . import that cannot be resolved
Lib/test/test_zipfile64.py
Imports undefined from test.support import TESTFN
Lib/unittest/test/test_program.py
Uses relative . import that cannot be resolved

Probably there are other problems, because I haven't checked:

Windows tests, because I don't have a Win machine
test_importlib, because it has a lot of failure (I don't know how to fix them yet)
Multiple other modules with slow tests
Modules with doctests which assert full __qualname__ with module names

To keep PRs reviews sane, I will include changes for several modules and split this big task of checking all test modules into several consecutive and rather simple pull requests.

The first one with the problems described above is on its way!

ethanfurman · 2022-01-18T18:59:16Z

I suggest your grouping of PRs be by error type -- so have batches of "relative import" fixes and batches of "TESTFN" fixes, etc.

serhiy-storchaka · 2022-01-18T19:28:36Z

New changeset 1292aa6 by Nikita Sobolev in branch 'main':
bpo-46425: Fix direct invocation of multiple test modules (GH-30666)
1292aa6

sobolevn · 2022-01-18T20:20:01Z

Thank you, Ethan. Good idea! Will do from the second PR.
Thanks for merging, Serhiy.

terryjreedy · 2022-01-18T20:31:31Z

Nikita, thanks for doing this.

Ethan's suggestion has two advantages. 1. It would fix most backport issues. By containing a mix of changes, PR-30666 cannot be backported as is. 'unittest.main' should always backport unless there is a context conflict. But 3.9 does not have os.helper and TESTFN is still in test.support in 3.9 and that change cannot backport. (I did not check 3.10.)

It restricts the knowledge needed to review (and be willing to merge). I could and would review and merge a patch with only unittest.main changes, but not, without checking the docs to learn about new changes, the mixture you posted.

I suggest you write a script to directly run each test.test_xyz file via subprocess and report to stdout. The test for this issue is a clean report. If you do so, I will run it on Windows in a fresh build of each of 3.9, 3.10, and 3.11.

Note that there are two possible commands: 'python -m test test_xyz' and 'python -m test.test_xyz'. They are subtly different. So I would add an option to do it each way.

With current 3.11, test_importlib runs fine on my Windows, with 1436 tests.

sobolevn · 2022-01-18T22:59:53Z

Terry, yes I am using this script to run tests locally:

#!/usr/bin/env bash

set -e

while read line; do 
  ./python.exe "$line"
done < files.txt

And this one to populate files.txt:

find ./Lib -iname 'test*.py' -o -iname '*tests.py' -exec echo {} >> files.txt \;

---

There are several other problems that I have to fix manually. For example, module-level raise unittest.case.SkipTest():

» ./python.exe Lib/test/test_winreg.py 
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/test/support/import_helper.py", line 77, in import_module
    return importlib.import_module(name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sobolev/Desktop/cpython/Lib/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1142, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'winreg'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_winreg.py", line 11, in <module>
    import_helper.import_module('winreg', required_on=['win'])
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sobolev/Desktop/cpython/Lib/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sobolev/Desktop/cpython/Lib/test/support/import_helper.py", line 26, in _ignore_deprecated_imports
    yield
    ^^^^^
  File "/Users/sobolev/Desktop/cpython/Lib/test/support/import_helper.py", line 81, in import_module
    raise unittest.SkipTest(str(msg))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
unittest.case.SkipTest: No module named 'winreg'

I am not sure what is the proper solution to this one 🤔

Probably, I will update my script for more advanced Python-based one and ignore this exception.

In the end, I expect to run this script without any failures. This is going to be a great result!

---

I will also check that all possible modes python -m test test_xyz / python -m test.test_xyz / python [Lib/test/test_xyz.py](/~https://github.com/python/cpython/blob/main/Lib/test/test_xyz.py) are supported.

---

Also, thank you for listing concrete problems with my first submission. The second one is bound to a single problem and a single submodule.

sobolevn · 2022-01-19T07:24:11Z

Important note, in practice there's also a difference in how one calls some test module:

python Lib/test/test_contextlib.py
python ./Lib/test/test_contextlib.py

Example of a test failure because of that: #30681

corona10 · 2022-01-20T01:43:58Z

New changeset 8105dd2 by Nikita Sobolev in branch '3.9':
[3.9] bpo-46425: Fix direct invocation of multiple test modules (GH-30666) (GH-30700)
8105dd2

corona10 · 2022-01-20T01:44:25Z

New changeset a6a0885 by Nikita Sobolev in branch '3.10':
[3.10] bpo-46425: Fix direct invocation of multiple test modules (GH-30666) (GH-30699)
a6a0885

serhiy-storchaka · 2022-01-21T07:36:37Z

New changeset 22f73bd by Nikita Sobolev in branch 'main':
bpo-46425: Fix direct invocation of test_contextlib (GH-30681)
22f73bd

asvetlov · 2022-01-22T11:06:31Z

New changeset 5a53400 by Nikita Sobolev in branch 'main':
bpo-46425: fix direct invocation of asyncio tests (bpo-30725)
5a53400

miss-islington · 2022-01-22T11:54:11Z

New changeset 3c4a374 by Miss Islington (bot) in branch '3.9':
bpo-46425: fix direct invocation of asyncio tests (GH-30725)
3c4a374

asvetlov · 2022-01-22T12:29:55Z

New changeset 6111d5d by Miss Islington (bot) in branch '3.10':
bpo-46425: fix direct invocation of asyncio tests (GH-30725) (bpo-30782)
6111d5d

asvetlov · 2022-01-22T16:03:17Z

New changeset 101a184 by Nikita Sobolev in branch 'main':
bpo-46425: fix direct invocation of test_traceback (GH-30746)
101a184

asvetlov · 2022-01-22T16:04:01Z

New changeset 55f4ec4 by Nikita Sobolev in branch 'main':
bpo-46425: use absolute imports in test_sqlite3 (GH-30676)
55f4ec4

asvetlov · 2022-01-22T16:05:09Z

New changeset 57316c5 by Nikita Sobolev in branch 'main':
bpo-46425: fix direct invocation of test_importlib (GH-30682)
57316c5

asvetlov · 2022-01-22T16:05:47Z

New changeset 1f8014c by Nikita Sobolev in branch 'main':
bpo-46425: fix direct invocation of test_fileutils and test_zoneinfo (GH-30792)
1f8014c

jaraco · 2022-01-23T02:38:46Z

New changeset d888ff5 by Jason R. Coombs in branch 'main':
bpo-46425: Partially revert "bpo-46425: fix direct invocation of test_importlib (GH-30682)" (GH-30799)
d888ff5

kumaraditya303 · 2022-04-16T12:13:55Z

@sobolevn Is this fixed now or are there more modules left?

sobolevn added 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error labels Jan 18, 2022

ezio-melotti transferred this issue from another repository Apr 10, 2022

kumaraditya303 closed this as completed May 11, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiple test modules fail to run if invoked directly #90583

Multiple test modules fail to run if invoked directly #90583

sobolevn commented Jan 18, 2022

sobolevn commented Jan 18, 2022

ethanfurman commented Jan 18, 2022

serhiy-storchaka commented Jan 18, 2022

sobolevn commented Jan 18, 2022

terryjreedy commented Jan 18, 2022

sobolevn commented Jan 18, 2022

sobolevn commented Jan 19, 2022

corona10 commented Jan 20, 2022

corona10 commented Jan 20, 2022

serhiy-storchaka commented Jan 21, 2022

asvetlov commented Jan 22, 2022

miss-islington commented Jan 22, 2022

asvetlov commented Jan 22, 2022

asvetlov commented Jan 22, 2022

asvetlov commented Jan 22, 2022

asvetlov commented Jan 22, 2022

asvetlov commented Jan 22, 2022

jaraco commented Jan 23, 2022

kumaraditya303 commented Apr 16, 2022

Multiple test modules fail to run if invoked directly #90583

Multiple test modules fail to run if invoked directly #90583

Comments

sobolevn commented Jan 18, 2022

sobolevn commented Jan 18, 2022

ethanfurman commented Jan 18, 2022

serhiy-storchaka commented Jan 18, 2022

sobolevn commented Jan 18, 2022

terryjreedy commented Jan 18, 2022

sobolevn commented Jan 18, 2022

sobolevn commented Jan 19, 2022

corona10 commented Jan 20, 2022

corona10 commented Jan 20, 2022

serhiy-storchaka commented Jan 21, 2022

asvetlov commented Jan 22, 2022

miss-islington commented Jan 22, 2022

asvetlov commented Jan 22, 2022

asvetlov commented Jan 22, 2022

asvetlov commented Jan 22, 2022

asvetlov commented Jan 22, 2022

asvetlov commented Jan 22, 2022

jaraco commented Jan 23, 2022

kumaraditya303 commented Apr 16, 2022