Skip to content

Commit

Permalink
gh-35322: Many more namespace packages
Browse files Browse the repository at this point in the history
    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes #1234" use "Introduce new method to
calculate 1+1"
-->
### 📚 Description
As discussed in
#35100 (comment), we
remove many of the empty `__init__.py` files, turning these packages
into PEP-420 implicit namespace packages.

The following packages will keep their `__init__.py` files because they
are intended to remain ordinary packages indefinitely:
- `sage.cpython`, `sage.structure` – because they are shipped as a whole
by **sagemath-objects**
- `sage.doctest`, `sage.repl` – because they are shipped as a whole by
**sagemath-repl**
- `sage.features` – because it is shipped as a whole by **sagemath-
environment**

<!-- Describe your changes here in detail -->
<!-- Why is this change required? What problem does it solve? -->
<!-- If it resolves an open issue, please link to the issue here. For
example "Closes #1337" -->

This is part of:
- #29705

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->

- [x] I have made sure that the title is self-explanatory and the
description concisely explains the PR.
- [x] I have linked an issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies
<!-- List all open pull requests that this PR logically depends on -->
<!--
- #xyz: short description why this is a dependency
- #abc: ...
-->
    
URL: #35322
Reported by: Matthias Köppe
Reviewer(s): Gonzalo Tornaría
  • Loading branch information
Release Manager committed Apr 4, 2023
2 parents ff95388 + dc8b3c5 commit 11cad42
Show file tree
Hide file tree
Showing 140 changed files with 152 additions and 15 deletions.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Empty file.
Empty file.
Empty file removed src/sage/combinat/sf/__init__.py
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file removed src/sage/databases/__init__.py
Empty file.
Empty file removed src/sage/dynamics/__init__.py
Empty file.
Empty file.
3 changes: 0 additions & 3 deletions src/sage/dynamics/cellular_automata/__init__.py

This file was deleted.

Empty file.
Empty file removed src/sage/functions/__init__.py
Empty file.
Empty file removed src/sage/game_theory/__init__.py
Empty file.
Empty file removed src/sage/games/__init__.py
Empty file.
Empty file removed src/sage/geometry/__init__.py
Empty file.
Empty file.
Empty file.
File renamed without changes.
Empty file.
Empty file.
File renamed without changes.
Empty file.
File renamed without changes.
Empty file.
Empty file.
Empty file removed src/sage/graphs/base/__init__.py
Empty file.
File renamed without changes.
Empty file.
File renamed without changes.
Empty file removed src/sage/groups/__init__.py
Empty file.
Empty file.
Empty file.
Empty file.
File renamed without changes.
Empty file.
File renamed without changes.
Empty file.
Empty file.
File renamed without changes.
Empty file.
Empty file.
File renamed without changes.
Empty file removed src/sage/homology/__init__.py
Empty file.
Empty file removed src/sage/interacts/__init__.py
Empty file.
Empty file removed src/sage/knots/__init__.py
Empty file.
Empty file removed src/sage/lfunctions/__init__.py
Empty file.
1 change: 0 additions & 1 deletion src/sage/libs/gap/__init__.py

This file was deleted.

Empty file removed src/sage/logic/__init__.py
Empty file.
Empty file removed src/sage/manifolds/__init__.py
Empty file.
Empty file.
File renamed without changes.
Empty file.
File renamed without changes.
Empty file.
File renamed without changes.
2 changes: 0 additions & 2 deletions src/sage/matroids/__init__.py

This file was deleted.

22 changes: 22 additions & 0 deletions src/sage/misc/cython.py
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,28 @@ def cython(filename, verbose=0, compile_message=False,
sage: cython('''
....: cdef size_t foo = 3/2
....: ''')
Check that Cython supports PEP 420 packages::
sage: cython('''
....: cimport sage.misc.cachefunc
....: ''')
sage: cython('''
....: from sage.misc.cachefunc cimport cache_key
....: ''')
In Cython 0.29.33 using `from PACKAGE cimport MODULE` is broken
when `PACKAGE` is a namespace package, see :trac:`35322`::
sage: cython('''
....: from sage.misc cimport cachefunc
....: ''')
Traceback (most recent call last):
...
RuntimeError: Error compiling Cython file:
...
...: 'sage/misc.pxd' not found
"""
if not filename.endswith('pyx'):
print("Warning: file (={}) should have extension .pyx".format(filename), file=sys.stderr)
Expand Down
4 changes: 2 additions & 2 deletions src/sage/misc/dev_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ def load_submodules(module=None, exclude_pattern=None):
load sage.geometry.polyhedron.palp_database... succeeded
load sage.geometry.polyhedron.ppl_lattice_polygon... succeeded
"""
import pkgutil
from .package_dir import walk_packages

if module is None:
import sage
Expand All @@ -181,7 +181,7 @@ def load_submodules(module=None, exclude_pattern=None):
else:
exclude = None

for importer, module_name, ispkg in pkgutil.walk_packages(module.__path__, module.__name__ + '.'):
for importer, module_name, ispkg in walk_packages(module.__path__, module.__name__ + '.'):
if ispkg or module_name in sys.modules:
continue

Expand Down
130 changes: 126 additions & 4 deletions src/sage/misc/package_dir.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@

import os
import glob
import sys
from contextlib import contextmanager


Expand Down Expand Up @@ -154,29 +155,29 @@ def is_package_or_sage_namespace_package_dir(path, *, distribution_filter=None):
:mod:`sage.cpython` is an ordinary package::
sage: from sage.misc.package_dir import is_package_or_sage_namespace_package_dir
sage: directory = os.path.dirname(sage.cpython.__file__); directory
sage: directory = sage.cpython.__path__[0]; directory
'.../sage/cpython'
sage: is_package_or_sage_namespace_package_dir(directory)
True
:mod:`sage.libs.mpfr` only has an ``__init__.pxd`` file, but we consider
it a package directory for consistency with Cython::
sage: directory = os.path.join(os.path.dirname(sage.libs.all.__file__), 'mpfr'); directory
sage: directory = os.path.join(sage.libs.__path__[0], 'mpfr'); directory
'.../sage/libs/mpfr'
sage: is_package_or_sage_namespace_package_dir(directory)
True
:mod:`sage` is designated to become an implicit namespace package::
sage: directory = os.path.dirname(sage.env.__file__); directory
sage: directory = sage.__path__[0]; directory
'.../sage'
sage: is_package_or_sage_namespace_package_dir(directory)
True
Not a package::
sage: directory = os.path.join(os.path.dirname(sage.symbolic.__file__), 'ginac'); directory
sage: directory = os.path.join(sage.symbolic.__path__[0], 'ginac'); directory
'.../sage/symbolic/ginac'
sage: is_package_or_sage_namespace_package_dir(directory)
False
Expand Down Expand Up @@ -211,3 +212,124 @@ def cython_namespace_package_support():
yield
finally:
Cython.Utils.is_package_dir = Cython.Build.Cythonize.is_package_dir = Cython.Build.Dependencies.is_package_dir = orig_is_package_dir


def walk_packages(path=None, prefix='', onerror=None):
r"""
Yield :class:`pkgutil.ModuleInfo` for all modules recursively on ``path``.
This version of the standard library function :func:`pkgutil.walk_packages`
addresses /~https://github.com/python/cpython/issues/73444 by handling
the implicit namespace packages in the package layout used by Sage;
see :func:`is_package_or_sage_namespace_package_dir`.
INPUT:
- ``path`` -- a list of paths to look for modules in or
``None`` (all accessible modules).
- ``prefix`` -- a string to output on the front of every module name
on output.
- ``onerror`` -- a function which gets called with one argument (the
name of the package which was being imported) if any exception
occurs while trying to import a package. If ``None``, ignore
:class:`ImportError` but propagate all other exceptions.
EXAMPLES::
sage: sorted(sage.misc.package_dir.walk_packages(sage.misc.__path__)) # a namespace package
[..., ModuleInfo(module_finder=FileFinder('.../sage/misc'), name='package_dir', ispkg=False), ...]
"""
# Adapted from /~https://github.com/python/cpython/blob/3.11/Lib/pkgutil.py

def iter_modules(path=None, prefix=''):
"""
Yield :class:`ModuleInfo` for all submodules on ``path``.
"""
from pkgutil import get_importer, iter_importers, ModuleInfo

if path is None:
importers = iter_importers()
elif isinstance(path, str):
raise ValueError("path must be None or list of paths to look for modules in")
else:
importers = map(get_importer, path)

yielded = {}
for i in importers:
for name, ispkg in iter_importer_modules(i, prefix):
if name not in yielded:
yielded[name] = 1
yield ModuleInfo(i, name, ispkg)

def iter_importer_modules(importer, prefix=''):
r"""
Yield :class:`ModuleInfo` for all modules of ``importer``.
"""
from importlib.machinery import FileFinder

if isinstance(importer, FileFinder):
if importer.path is None or not os.path.isdir(importer.path):
return

yielded = {}
import inspect
try:
filenames = os.listdir(importer.path)
except OSError:
# ignore unreadable directories like import does
filenames = []
filenames.sort() # handle packages before same-named modules

for fn in filenames:
modname = inspect.getmodulename(fn)
if modname and (modname in ['__init__', 'all']
or modname.startswith('all__')
or modname in yielded):
continue

path = os.path.join(importer.path, fn)
ispkg = False

if not modname and os.path.isdir(path) and '.' not in fn:
modname = fn
if not (ispkg := is_package_or_sage_namespace_package_dir(path)):
continue

if modname and '.' not in modname:
yielded[modname] = 1
yield prefix + modname, ispkg

elif not hasattr(importer, 'iter_modules'):
yield from []

else:
yield from importer.iter_modules(prefix)

def seen(p, m={}):
if p in m:
return True
m[p] = True

for info in iter_modules(path, prefix):
yield info

if info.ispkg:
try:
__import__(info.name)
except ImportError:
if onerror is not None:
onerror(info.name)
except Exception:
if onerror is not None:
onerror(info.name)
else:
raise
else:
path = getattr(sys.modules[info.name], '__path__', None) or []

# don't traverse path items we've seen before
path = [p for p in path if not seen(p)]

yield from walk_packages(path, info.name + '.', onerror)
Empty file removed src/sage/modular/__init__.py
Empty file.
Empty file removed src/sage/modular/abvar/__init__.py
Empty file.
Empty file.
Empty file.
Empty file removed src/sage/modular/hecke/__init__.py
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
1 change: 0 additions & 1 deletion src/sage/modular/quasimodform/__init__.py

This file was deleted.

Empty file.
Empty file removed src/sage/modular/ssmod/__init__.py
Empty file.
Empty file removed src/sage/modules/__init__.py
Empty file.
Empty file.
File renamed without changes.
Empty file.
File renamed without changes.
Empty file.
File renamed without changes.
Empty file removed src/sage/monoids/__init__.py
Empty file.
Empty file removed src/sage/parallel/__init__.py
Empty file.
Empty file removed src/sage/plot/__init__.py
Empty file.
Empty file removed src/sage/plot/plot3d/__init__.py
Empty file.
Empty file removed src/sage/probability/__init__.py
Empty file.
Empty file.
Empty file.
Empty file removed src/sage/quivers/__init__.py
Empty file.
File renamed without changes.
Empty file.
Empty file removed src/sage/rings/convert/__init__.py
Empty file.
File renamed without changes.
Empty file.
Empty file.
File renamed without changes.
Empty file.
Empty file removed src/sage/rings/padics/__init__.py
Empty file.
Empty file.
File renamed without changes.
Empty file.
Empty file.
Empty file.
Empty file removed src/sage/sandpiles/__init__.py
Empty file.
Empty file removed src/sage/sat/__init__.py
Empty file.
Empty file removed src/sage/schemes/__init__.py
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
File renamed without changes.
Empty file removed src/sage/schemes/toric/__init__.py
Empty file.
Empty file.
File renamed without changes.
Empty file removed src/sage/server/__init__.py
Empty file.
Empty file removed src/sage/stats/__init__.py
Empty file.
Empty file.
File renamed without changes.
Empty file removed src/sage/stats/hmm/__init__.py
Empty file.
Empty file removed src/sage/symbolic/__init__.py
Empty file.
Empty file.
File renamed without changes.
4 changes: 2 additions & 2 deletions src/sage/symbolic/pynac.pxi
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ Declarations for pynac, a Python frontend for ginac
Check that we can externally cimport this (:trac:`18825`)::
sage: cython( # long time; random compiler warnings # optional - sage.misc.cython
sage: cython( # optional - sage.misc.cython
....: '''
....: from sage.symbolic cimport expression
....: cimport sage.symbolic.expression
....: ''')
"""

Expand Down
Empty file removed src/sage/tensor/__init__.py
Empty file.
Empty file.
Empty file removed src/sage/topology/__init__.py
Empty file.
Empty file removed src/sage/typeset/__init__.py
Empty file.

0 comments on commit 11cad42

Please sign in to comment.