Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-101659: Isolate "obmalloc" State to Each Interpreter #101660

Merged
merged 58 commits into from
Apr 24, 2023
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
07a09d4
Pass PyInterpreterState to pymalloc_*().
ericsnowcurrently Oct 6, 2022
ca75048
Move the object arenas to the interpreter state.
ericsnowcurrently Oct 7, 2022
4ee199b
Drop an errant #define.
ericsnowcurrently Feb 7, 2023
2768fa4
Leave dump_debug_stats in the global state.
ericsnowcurrently Feb 7, 2023
bf9425f
Dynamically initialize obmalloc for subinterpreters.
ericsnowcurrently Feb 9, 2023
d5da34b
Merge branch 'main' into per-interpreter-alloc
ericsnowcurrently Mar 9, 2023
6c3111c
Pass around struct _obmalloc_state* instead of PyInterpeterState*.
ericsnowcurrently Mar 8, 2023
4dc087d
Add _PyInterpreterConfig.use_main_obmalloc.
ericsnowcurrently Mar 9, 2023
1ae33a0
Add a comment about why per-interpreter obmalloc requires multi-phase…
ericsnowcurrently Mar 9, 2023
5b54d63
Add a TODO comment.
ericsnowcurrently Mar 9, 2023
9f4f8f3
Optionally use the main interpreter's obmalloc state.
ericsnowcurrently Mar 9, 2023
aa10204
Pass use_main_obmalloc to run_in_subinterp() in test_import.
ericsnowcurrently Mar 9, 2023
69d9a2d
_Py_GetAllocatedBlocks() -> _Py_GetGlobalAllocatedBlocks().
ericsnowcurrently Mar 10, 2023
25378f8
Errors from _Py_NewInterpreterFromConfig() are no longer fatal.
ericsnowcurrently Mar 10, 2023
1c5b109
Chain the exceptions.
ericsnowcurrently Mar 13, 2023
f36426b
Swap out the failed tstate.
ericsnowcurrently Mar 10, 2023
54b9f09
Remaining static builtin types must be fixed.
ericsnowcurrently Mar 13, 2023
2358a42
Add PyInterpreterState.sysdict_copy.
ericsnowcurrently Mar 13, 2023
b6502e1
Set m_copy to None for sys and builtins.
ericsnowcurrently Mar 13, 2023
678e67b
Add _PyIO_InitTypes().
ericsnowcurrently Mar 13, 2023
69a5829
Fix test_capi.
ericsnowcurrently Mar 13, 2023
3feb408
Avoid allocation for shared exceptions.
ericsnowcurrently Mar 13, 2023
05806fc
Fix the ChannelID tp_name.
ericsnowcurrently Mar 13, 2023
b1cd7bb
Merge branch 'main' into per-interpreter-alloc
ericsnowcurrently Mar 29, 2023
4feb2b7
Do not include the total from interpreters sharing with main.
ericsnowcurrently Mar 29, 2023
136ad2f
Add _PyRuntime.obmalloc.interpreter_leaks.
ericsnowcurrently Mar 29, 2023
e19bb37
Track leaked blocks across init/fini cycles.
ericsnowcurrently Mar 29, 2023
6c51997
Clean up assumptions around runtime fini.
ericsnowcurrently Mar 29, 2023
f0fcaf6
Merge branch 'main' into per-interpreter-alloc
ericsnowcurrently Mar 29, 2023
0ff65ff
Add stubs for when WITH_PYMALLOC isn't defined.
ericsnowcurrently Mar 30, 2023
7db8d4a
Decref the key in the right interpreter in _extensions_cache_set().
ericsnowcurrently Mar 31, 2023
38bee89
Don't test against sys (for now).
ericsnowcurrently Mar 31, 2023
375a8f2
Clean up SubinterpImportTests.
ericsnowcurrently Mar 31, 2023
b0a9e11
Ensure we are testing against the right type of extension.
ericsnowcurrently Mar 31, 2023
5e5d5d5
Add a test that uses an isolated interpreter.
ericsnowcurrently Mar 31, 2023
25809ce
Fix is_core_module().
ericsnowcurrently Apr 4, 2023
616d3dd
Merge branch 'main' into per-interpreter-alloc
ericsnowcurrently Apr 4, 2023
43a836b
Ignore last_final_leaks.
ericsnowcurrently Apr 4, 2023
1841b55
Fix a typo.
ericsnowcurrently Apr 4, 2023
299527e
Merge branch 'main' into per-interpreter-alloc
ericsnowcurrently Apr 4, 2023
0091e48
Add a note about global state owned by the module.
ericsnowcurrently Apr 5, 2023
9f74f7b
Factor out GLOBAL_MALLOC() and GLOBAL_FREE().
ericsnowcurrently Apr 5, 2023
10c3589
Switch to the raw allocator.
ericsnowcurrently Apr 5, 2023
ff727ec
Merge branch 'channels-raw-allocator' into per-interpreter-alloc
ericsnowcurrently Apr 5, 2023
593430b
Use the raw allocator for _PyCrossInterpreterData_InitWithSize().
ericsnowcurrently Apr 5, 2023
f5ae710
atexit_callback -> atexit_py_callback.
ericsnowcurrently Apr 5, 2023
e6d4776
Add pycore_atexit.h.
ericsnowcurrently Apr 5, 2023
c719f02
Add _Py_AtExit().
ericsnowcurrently Apr 5, 2023
47c302d
Add a TODO comment.
ericsnowcurrently Apr 5, 2023
aaeaaa6
Move _Py_AtExit() to the public API.
ericsnowcurrently Apr 5, 2023
b5396e4
Test a constraint.
ericsnowcurrently Apr 5, 2023
448b48a
Add an atexit callback for _xxinterpchannels.
ericsnowcurrently Apr 5, 2023
c86f738
Implement the callback.
ericsnowcurrently Apr 5, 2023
1827feb
Drop the _PyCrossInterpreterData_Clear() call in _xxinterpchannels.
ericsnowcurrently Apr 5, 2023
82b395c
Drop the _PyCrossInterpreterData_Clear() call in _xxsubinterpreters.
ericsnowcurrently Apr 5, 2023
df77a64
Merge branch 'atexit-c-callback' into per-interpreter-alloc
ericsnowcurrently Apr 6, 2023
22758a3
Merge branch 'main' into per-interpreter-alloc
ericsnowcurrently Apr 6, 2023
0fd74a9
Merge branch 'main' into per-interpreter-alloc
ericsnowcurrently Apr 24, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion Include/internal/pycore_interp.h
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,9 @@ extern "C" {
#include "pycore_function.h" // FUNC_MAX_WATCHERS
#include "pycore_genobject.h" // struct _Py_async_gen_state
#include "pycore_gc.h" // struct _gc_runtime_state
#include "pycore_list.h" // struct _Py_list_state
#include "pycore_global_objects.h" // struct _Py_interp_static_objects
#include "pycore_list.h" // struct _Py_list_state
#include "pycore_obmalloc.h" // struct obmalloc_state
#include "pycore_tuple.h" // struct _Py_tuple_state
#include "pycore_typeobject.h" // struct type_cache
#include "pycore_unicodeobject.h" // struct _Py_unicode_state
Expand Down Expand Up @@ -89,6 +90,8 @@ struct _is {
int _initialized;
int finalizing;

struct _obmalloc_state obmalloc;

struct _ceval_state ceval;
struct _gc_runtime_state gc;

Expand Down
5 changes: 4 additions & 1 deletion Include/internal/pycore_obmalloc.h
Original file line number Diff line number Diff line change
Expand Up @@ -657,8 +657,11 @@ struct _obmalloc_usage {
#endif /* WITH_PYMALLOC_RADIX_TREE */


struct _obmalloc_state {
struct _obmalloc_global_state {
int dump_debug_stats;
};

struct _obmalloc_state {
struct _obmalloc_pools pools;
struct _obmalloc_mgmt mgmt;
struct _obmalloc_usage usage;
Expand Down
6 changes: 5 additions & 1 deletion Include/internal/pycore_obmalloc_init.h
Original file line number Diff line number Diff line change
Expand Up @@ -54,9 +54,13 @@ extern "C" {
# error "NB_SMALL_SIZE_CLASSES should be less than 64"
#endif

#define _obmalloc_state_INIT(obmalloc) \
#define _obmalloc_global_state_INIT \
{ \
.dump_debug_stats = -1, \
}

#define _obmalloc_state_INIT(obmalloc) \
{ \
.pools = { \
.used = _obmalloc_pools_INIT(obmalloc.pools), \
}, \
Expand Down
3 changes: 1 addition & 2 deletions Include/internal/pycore_runtime.h
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ extern "C" {
#include "pycore_pymem.h" // struct _pymem_allocators
#include "pycore_pyhash.h" // struct pyhash_runtime_state
#include "pycore_pythread.h" // struct _pythread_runtime_state
#include "pycore_obmalloc.h" // struct obmalloc_state
#include "pycore_signal.h" // struct _signals_runtime_state
#include "pycore_time.h" // struct _time_runtime_state
#include "pycore_tracemalloc.h" // struct _tracemalloc_runtime_state
Expand Down Expand Up @@ -88,7 +87,7 @@ typedef struct pyruntimestate {
_Py_atomic_address _finalizing;

struct _pymem_allocators allocators;
struct _obmalloc_state obmalloc;
struct _obmalloc_global_state obmalloc;
struct pyhash_runtime_state pyhash_state;
struct _time_runtime_state time;
struct _pythread_runtime_state threads;
Expand Down
7 changes: 4 additions & 3 deletions Include/internal/pycore_runtime_init.h
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ extern "C" {
_pymem_allocators_debug_INIT, \
_pymem_allocators_obj_arena_INIT, \
}, \
.obmalloc = _obmalloc_state_INIT(runtime.obmalloc), \
.obmalloc = _obmalloc_global_state_INIT, \
.pyhash_state = pyhash_state_INIT, \
.signals = _signals_RUNTIME_INIT, \
.interpreters = { \
Expand Down Expand Up @@ -94,7 +94,7 @@ extern "C" {
}, \
}, \
}, \
._main_interpreter = _PyInterpreterState_INIT, \
._main_interpreter = _PyInterpreterState_INIT(runtime._main_interpreter), \
}

#ifdef HAVE_DLOPEN
Expand All @@ -110,10 +110,11 @@ extern "C" {
# define DLOPENFLAGS_INIT
#endif

#define _PyInterpreterState_INIT \
#define _PyInterpreterState_INIT(interp) \
{ \
.id_refcount = -1, \
DLOPENFLAGS_INIT \
.obmalloc = _obmalloc_state_INIT(interp.obmalloc), \
.ceval = { \
.recursion_limit = Py_DEFAULT_RECURSION_LIMIT, \
}, \
Expand Down
Loading