multiprocessing tests log: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown

[vstinner]

BPO	45209
Nosy	@vstinner, @ambv, @miss-islington, @sobolevn
PRs	bpo-45209: fix UserWarning: resource_tracker in test_multiprocessing #28377 [3.10] bpo-45209: fix UserWarning: resource_tracker in test_multiprocessing (GH-28377) #28499 [3.9] bpo-45209: fix UserWarning: resource_tracker in test_multiprocessing (GH-28377) #28500

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-09-21.18:37:39.102>
created_at = <Date 2021-09-15.13:29:13.571>
labels = ['tests', '3.9', '3.10', '3.11']
title = 'multiprocessing tests log: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown'
updated_at = <Date 2021-09-21.18:37:39.101>
user = '/~https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-09-21.18:37:39.101>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-09-21.18:37:39.102>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2021-09-15.13:29:13.571>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 45209
keywords = ['patch']
message_count = 7.0
messages = ['401837', '401866', '401902', '402336', '402341', '402342', '402343']
nosy_count = 4.0
nosy_names = ['vstinner', 'lukasz.langa', 'miss-islington', 'sobolevn']
pr_nums = ['28377', '28499', '28500']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue45209'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[vstinner]

0:03:25 load avg: 12.33 [250/427] test_multiprocessing_forkserver passed (...)
/home/vstinner/python/main/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
/home/vstinner/python/main/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/psm_33506d9a': [Errno 2] No such file or directory: '/psm_33506d9a'
warnings.warn('resource_tracker: %r: %s' % (name, e))

[sobolevn]

I would like to work on this, if no one has started yet.

[sobolevn]

The problem was in test_shared_memory_cleaned_after_process_termination test.

» ./python.exe -m test -v test_multiprocessing_forkserver -m test_shared_memory_cleaned_after_process_termination
== CPython 3.11.0a0 (heads/main:09b4ad11f3, Sep 15 2021, 20:50:50) [Clang 11.0.0 (clang-1100.0.33.16)]
== macOS-10.14.6-x86_64-i386-64bit little-endian
== cwd: /Users/sobolev/Desktop/cpython/build/test_python_14874æ
== CPU count: 4
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 1.98 Run tests sequentially
0:00:00 load avg: 1.98 [1/1] test_multiprocessing_forkserver
test_shared_memory_cleaned_after_process_termination (test.test_multiprocessing_forkserver.WithProcessesTestSharedMemory) ... ok
/Users/sobolev/Desktop/cpython/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/Users/sobolev/Desktop/cpython/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/psm_67480e45': [Errno 2] No such file or directory: '/psm_67480e45'
  warnings.warn('resource_tracker: %r: %s' % (name, e))

----------------------------------------------------------------------
Ran 1 test in 0.775s

OK

== Tests result: SUCCESS ==

1 test OK.

Total duration: 1.6 sec
Tests result: SUCCESS

My patch (location

cpython/Lib/test/_test_multiprocessing.py

Line 4179 in 51056b4

):

resource_tracker.unregister(f"/{name}", "shared_memory")

With this change no warning is generated:

» ./python.exe -m test -v test_multiprocessing_forkserver -m test_shared_memory_cleaned_after_process_termination
== CPython 3.11.0a0 (heads/main:09b4ad11f3, Sep 15 2021, 20:50:50) [Clang 11.0.0 (clang-1100.0.33.16)]
== macOS-10.14.6-x86_64-i386-64bit little-endian
== cwd: /Users/sobolev/Desktop/cpython/build/test_python_24281æ
== CPU count: 4
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 1.80 Run tests sequentially
0:00:00 load avg: 1.80 [1/1] test_multiprocessing_forkserver
test_shared_memory_cleaned_after_process_termination (test.test_multiprocessing_forkserver.WithProcessesTestSharedMemory) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.732s

OK

== Tests result: SUCCESS ==

1 test OK.

Total duration: 1.5 sec
Tests result: SUCCESS

My other highly-related PR where I refactor several SharedMemory tests: #28294

[ambv]

New changeset f604cf1 by Nikita Sobolev in branch 'main':
bpo-45209: fix UserWarning: resource_tracker in test_multiprocessing (GH-28377)
f604cf1

[miss-islington]

New changeset 09e5016 by Miss Islington (bot) in branch '3.10':
bpo-45209: fix UserWarning: resource_tracker in test_multiprocessing (GH-28377)
09e5016

[vstinner]

New changeset 65268ab by Miss Islington (bot) in branch '3.9':
bpo-45209: fix UserWarning: resource_tracker in test_multiprocessing (GH-28377) (GH-28500)
65268ab

[vstinner]

Thanks Nikita Sobolev for the fix.

[guglovich]

On 3.12 on Arch Linux, this error is repeated.

[vstinner]

Please open a new issue wirh a reproducer.