Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose _Py_NewInterpreter() as Py_NewInterpreterFromConfig() #98608

Closed
ericsnowcurrently opened this issue Oct 24, 2022 · 0 comments
Closed

Expose _Py_NewInterpreter() as Py_NewInterpreterFromConfig() #98608

ericsnowcurrently opened this issue Oct 24, 2022 · 0 comments
Assignees
Labels
3.12 bugs and security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-subinterpreters type-feature A feature request or enhancement

Comments

@ericsnowcurrently
Copy link
Member

ericsnowcurrently commented Oct 24, 2022

A while back I added _Py_NewInterpreter() (a "private" API) to support configuring the new interpreter. Ultimately, I'd like to adjust
the signature a little and then make the function part of the public API (as Py_NewInterpreterFromConfig()).

My plan:

  1. change the argument to a new _PyInterpreterConfig struct
  2. rename the function to Py_NewInterpreterFromConfig(), inspired by Py_InitializeFromConfig() (takes a PyInterpreterConfig instead of isolated_subinterpreter)
  3. split up the boolean isolated_subinterpreter into the corresponding multiple granular settings
    • allow_fork
    • allow_subprocess
    • allow_threads
  4. drop PyConfig._isolated_interpreter

Note that the current default (Py_NewInterpeter() and Py_Initialize*()) allows fork, subprocess, and threads, and the optional "isolated" interpreter disables all three. I'm not planning on changing any of that here.

My main objective here is to expose the existing API in a way that we can do the following afterward:

  • stop giving the option to disallow subprocess (i.e. drop PyInterpreterConfig.allow_subprocess)
  • add an option to disallow just "exec" instead
  • stop disallowing threads as a default behavior for an "isolated" interpreter (we'd still keep the option though)
  • add the option to disallow daemon threads
  • add an option to check if each extension supports running in multiple interpreters
  • add other options for PEP 684 (per-interpreter GIL)

Linked PRs

@ericsnowcurrently ericsnowcurrently added type-feature A feature request or enhancement interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-subinterpreters 3.12 bugs and security fixes labels Oct 24, 2022
@ericsnowcurrently ericsnowcurrently self-assigned this Oct 25, 2022
ericsnowcurrently added a commit that referenced this issue Oct 26, 2022
…() (gh-98609)

(see #98608)

This change does the following:

1. change the argument to a new `_PyInterpreterConfig` struct
2. rename the function to `_Py_NewInterpreterFromConfig()`, inspired by `Py_InitializeFromConfig()` (takes a `_PyInterpreterConfig`  instead of `isolated_subinterpreter`)
3. split up the boolean `isolated_subinterpreter` into the corresponding multiple granular settings
   * allow_fork
   * allow_subprocess
   * allow_threads
4. add `PyInterpreterState.feature_flags` to store those settings
5. add a function for checking if a feature is enabled on an opaque `PyInterpreterState *`
6. drop `PyConfig._isolated_interpreter`

The existing default (see `Py_NewInterpeter()` and `Py_Initialize*()`) allows fork, subprocess, and threads and the optional "isolated" interpreter (see the `_xxsubinterpreters` module) disables all three.  None of that changes here; the defaults are preserved.

Note that the given `_PyInterpreterConfig` will not be used outside `_Py_NewInterpreterFromConfig()`, nor preserved.  This contrasts with how `PyConfig` is currently preserved, used, and even modified outside `Py_InitializeFromConfig()`.  I'd rather just avoid that mess from the start for `_PyInterpreterConfig`.  We can preserve it later if we find an actual need.

This change allows us to follow up with a number of improvements (e.g. stop disallowing subprocess and support disallowing exec instead).

(Note that this PR adds "private" symbols.  We'll probably make them public, and add docs, in a separate change.)
Repository owner moved this from In Review to Done in Fancy CPython Board Oct 26, 2022
ericsnowcurrently added a commit that referenced this issue Mar 21, 2023
…) as Fatal (gh-102657)

Prior to this change, errors in _Py_NewInterpreterFromConfig() were always fatal.  Instead, callers should be able to handle such errors and keep going.  That's what this change supports.  (This was an oversight in the original implementation of _Py_NewInterpreterFromConfig().)  Note that the existing [fatal] behavior of the public Py_NewInterpreter() is preserved.

#98608
ericsnowcurrently added a commit that referenced this issue Mar 21, 2023
The error-handling code in new_interpreter() has been broken for a while.  We hadn't noticed because those code mostly doesn't fail.  (I noticed while working on gh-101660.)  The problem is that we try to clear/delete the newly-created thread/interpreter using itself, which just failed.  The solution is to switch back to the calling thread state first.

#98608
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Mar 21, 2023
…02658)

The error-handling code in new_interpreter() has been broken for a while.  We hadn't noticed because those code mostly doesn't fail.  (I noticed while working on pythongh-101660.)  The problem is that we try to clear/delete the newly-created thread/interpreter using itself, which just failed.  The solution is to switch back to the calling thread state first.

(cherry picked from commit d1b883b)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
https: //github.com/python/issues/98608
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Mar 21, 2023
…02658)

The error-handling code in new_interpreter() has been broken for a while.  We hadn't noticed because those code mostly doesn't fail.  (I noticed while working on pythongh-101660.)  The problem is that we try to clear/delete the newly-created thread/interpreter using itself, which just failed.  The solution is to switch back to the calling thread state first.

(cherry picked from commit d1b883b)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
https: //github.com/python/issues/98608
miss-islington added a commit that referenced this issue Mar 21, 2023
The error-handling code in new_interpreter() has been broken for a while.  We hadn't noticed because those code mostly doesn't fail.  (I noticed while working on gh-101660.)  The problem is that we try to clear/delete the newly-created thread/interpreter using itself, which just failed.  The solution is to switch back to the calling thread state first.

(cherry picked from commit d1b883b)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
https: //github.com//issues/98608
miss-islington added a commit that referenced this issue Mar 21, 2023
The error-handling code in new_interpreter() has been broken for a while.  We hadn't noticed because those code mostly doesn't fail.  (I noticed while working on gh-101660.)  The problem is that we try to clear/delete the newly-created thread/interpreter using itself, which just failed.  The solution is to switch back to the calling thread state first.

(cherry picked from commit d1b883b)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
https: //github.com//issues/98608
Fidget-Spinner pushed a commit to Fidget-Spinner/cpython that referenced this issue Mar 27, 2023
…onfig() as Fatal (pythongh-102657)

Prior to this change, errors in _Py_NewInterpreterFromConfig() were always fatal.  Instead, callers should be able to handle such errors and keep going.  That's what this change supports.  (This was an oversight in the original implementation of _Py_NewInterpreterFromConfig().)  Note that the existing [fatal] behavior of the public Py_NewInterpreter() is preserved.

python#98608
Fidget-Spinner pushed a commit to Fidget-Spinner/cpython that referenced this issue Mar 27, 2023
…02658)

The error-handling code in new_interpreter() has been broken for a while.  We hadn't noticed because those code mostly doesn't fail.  (I noticed while working on pythongh-101660.)  The problem is that we try to clear/delete the newly-created thread/interpreter using itself, which just failed.  The solution is to switch back to the calling thread state first.

python#98608
warsaw pushed a commit to warsaw/cpython that referenced this issue Apr 11, 2023
…onfig() as Fatal (pythongh-102657)

Prior to this change, errors in _Py_NewInterpreterFromConfig() were always fatal.  Instead, callers should be able to handle such errors and keep going.  That's what this change supports.  (This was an oversight in the original implementation of _Py_NewInterpreterFromConfig().)  Note that the existing [fatal] behavior of the public Py_NewInterpreter() is preserved.

python#98608
warsaw pushed a commit to warsaw/cpython that referenced this issue Apr 11, 2023
…02658)

The error-handling code in new_interpreter() has been broken for a while.  We hadn't noticed because those code mostly doesn't fail.  (I noticed while working on pythongh-101660.)  The problem is that we try to clear/delete the newly-created thread/interpreter using itself, which just failed.  The solution is to switch back to the calling thread state first.

python#98608
vstinner added a commit to vstinner/cpython that referenced this issue Jul 24, 2023
Move PyInterpreterConfig structure and associated macros from
initconfig.h to pylifecycle.h: it's not related to the Python
Initialization Configuration.
vstinner added a commit that referenced this issue Jul 24, 2023
Move PyInterpreterConfig structure and associated macros from
initconfig.h to pylifecycle.h: it's not related to the Python
Initialization Configuration.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jul 24, 2023
…07191)

Move PyInterpreterConfig structure and associated macros from
initconfig.h to pylifecycle.h: it's not related to the Python
Initialization Configuration.
(cherry picked from commit e717b47)

Co-authored-by: Victor Stinner <vstinner@python.org>
vstinner added a commit that referenced this issue Jul 24, 2023
#107198)

gh-98608: Move PyInterpreterConfig to pylifecycle.h (GH-107191)

Move PyInterpreterConfig structure and associated macros from
initconfig.h to pylifecycle.h: it's not related to the Python
Initialization Configuration.
(cherry picked from commit e717b47)

Co-authored-by: Victor Stinner <vstinner@python.org>
jtcave pushed a commit to jtcave/cpython that referenced this issue Jul 27, 2023
)

Move PyInterpreterConfig structure and associated macros from
initconfig.h to pylifecycle.h: it's not related to the Python
Initialization Configuration.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.12 bugs and security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-subinterpreters type-feature A feature request or enhancement
Projects
Status: Done
Development

No branches or pull requests

1 participant