diff --git a/CinderDoc/strict_modules/DOCS b/CinderDoc/strict_modules/DOCS new file mode 100644 index 00000000000..7a73fa7c7be --- /dev/null +++ b/CinderDoc/strict_modules/DOCS @@ -0,0 +1,6 @@ +[strict_modules] + type = WIKI + srcs = [ + glob(*.rst) + ] + wiki_root_path = Python/Cinder/External_Public/Strict_Modules diff --git a/CinderDoc/strict_modules/guide/conversion/class_inst_conflict.rst b/CinderDoc/strict_modules/guide/conversion/class_inst_conflict.rst new file mode 100644 index 00000000000..03b6fd0960d --- /dev/null +++ b/CinderDoc/strict_modules/guide/conversion/class_inst_conflict.rst @@ -0,0 +1,48 @@ +Class / Instance Conflict +######################### + + +One of the changes that strict modules introduces is the promotion of instance +members to being class level declarations. For more information on this pattern +see :doc:`../limitations/class_attrs`. + +A typical case for this is when you'd like to have a default method implementation +but override it on a per instance basis: + +.. code-block:: python + + class C: + def f(self): + return 42 + + a = C() + a.f = lambda: "I'm a special snowflake" + + +If you attempt this inside of a strict module you'll get an AttributeError that +says "'C' object attribute 'f' is read-only". This is because the instance +doesn't have any place to store the method. You might think that you can declare +the field explicitly as specified in the documentation: + +.. code-block:: python + + class C: + f: ... + def f(self): + return 42 + +But instead you'll get an error reported by strict modules stating that there's +a conflict with the variable. To get around this issue you can promote the function +to always be treated as an instance member: + +.. code-block:: python + + class C: + def __init__(self): + self.f = self.default_f + + def default_f(self): + return 42 + + a = C() + a.f = lambda: "I'm a special snowflake" # Ok, you are a special snowflake diff --git a/CinderDoc/strict_modules/guide/conversion/external_modification.rst b/CinderDoc/strict_modules/guide/conversion/external_modification.rst new file mode 100644 index 00000000000..826b0245200 --- /dev/null +++ b/CinderDoc/strict_modules/guide/conversion/external_modification.rst @@ -0,0 +1,53 @@ +Modifying External State +######################## + +Strict modules enforces object :doc:`ownership `, +and will not allow module-level code to modify any object defined +in a different module. + +One common example of this is to have a global registry of some sort of +objects: + +**methods.py** + +.. code-block:: python + + import __strict__ + + ROUTES = list() + + def route(f): + ROUTES.append(f) + return f + +**routes.py** + +.. code-block:: python + + import __strict__ + + from methods import route + + @route + def create_user(*args): + ... + + +Here we have one module which is maintaining a global registry, which is +populated as a side effect of importing another module. If for some reason +one module doesn't get imported or if the order of imports changes then the +program's execution can change. When strict modules analyzes this code it will +report a :doc:`/guide/errors/modify_imported_value`. + +A better pattern for this is to explicitly register the values in a central +location: + +**methods.py** + +.. code-block:: python + + import __strict__ + + from routes import create_user + + ROUTES = [create_user, ...] diff --git a/CinderDoc/strict_modules/guide/conversion/index.rst b/CinderDoc/strict_modules/guide/conversion/index.rst new file mode 100644 index 00000000000..fd90b9d28e4 --- /dev/null +++ b/CinderDoc/strict_modules/guide/conversion/index.rst @@ -0,0 +1,12 @@ +Conversion Tips +############### + +This section of the documentation includes common patterns that violate the +limitations of strict modules and solutions you can use to work around them. + +.. toctree:: + :maxdepth: 1 + :titlesonly: + :glob: + + * diff --git a/CinderDoc/strict_modules/guide/conversion/loose_slots.rst b/CinderDoc/strict_modules/guide/conversion/loose_slots.rst new file mode 100644 index 00000000000..86109da63d7 --- /dev/null +++ b/CinderDoc/strict_modules/guide/conversion/loose_slots.rst @@ -0,0 +1,42 @@ +The @loose_slots decorator +########################## + +Instances of strict classes have `__slots__ +`_ automatically +created for them. This means they will raise ``AttributeError`` if you try to +add any attribute to them that isn't declared with a type annotation on the +class itself (e.g. ``attrname: int``) or assigned in the ``__init__`` method. + +When initially converting a module to strict, if it is widely-used it can be +hard to verify that there isn't code somewhere tacking extra attributes onto +instances of classes defined in that module. In this case, you can temporarily +place the ``strict_modules.loose_slots`` decorator on the class for a safer +transition. Example: + +.. code-block:: python + + import __strict__ + + from compiler.strict.runtime import loose_slots + + @loose_slots + class MyClass: + ... + +This decorator will allow extra attributes to be added to the class, but will +fire a warning when it happens. You can access these warnings by setting a +warnings callback function: + +.. code-block:: python + + from cinder import cinder_set_warnings_handler + + def log_cinder_warning(msg: str, *args: object) -> None: + # ... + + cinder_set_warnings_handler(log_cinder_warning) + +Typically you'd want to set a warnings handler that logs these warnings somewhere, +then you can deploy some new strict modules using `@loose_slots`, +and once the code has been in production for a bit and you see no warnings +fired, you can safely remove `@loose_slots`. diff --git a/CinderDoc/strict_modules/guide/conversion/module_access.rst b/CinderDoc/strict_modules/guide/conversion/module_access.rst new file mode 100644 index 00000000000..17951e1e772 --- /dev/null +++ b/CinderDoc/strict_modules/guide/conversion/module_access.rst @@ -0,0 +1,27 @@ +Top-level Module Access +####################### + +A common pattern is to import a module and access members from that module: + +.. code-block:: python + + from useful import submodule + + class MyClass(submodule.BaseClass): + pass + +If “submodule” is not strict, then we don't know what it is and what side +effects could happen by dotting through it. So this pattern is disallowed +inside of a strict module when importing from a non-strict module. Instead +you can transform the code to: + +.. code-block:: python + + from useful.submodule import BaseClass + + class MyClass(BaseClass): + pass + +This will cause any side effects that are possible to occur only when +the non-strict module is imported; the execution of the rest of the +strict module will be known to be side effect free. diff --git a/CinderDoc/strict_modules/guide/conversion/singletons.rst b/CinderDoc/strict_modules/guide/conversion/singletons.rst new file mode 100644 index 00000000000..1b6e29424b4 --- /dev/null +++ b/CinderDoc/strict_modules/guide/conversion/singletons.rst @@ -0,0 +1,66 @@ +Global Singletons +################# + +Sometimes it might be useful to encapsulate a set of functionality into +a class and then have a global singleton of that class. And sometimes +that global singleton might have dependencies on non-strict code which +makes it impossible to construct at the top-level in a strict module. + +.. code-block:: python + + from non_strict import get_counter_start + + class Counter: + def __init__(self) -> None: + self.value: int = get_counter_start() + + def next(self) -> int: + res = self.value + self.value += 1 + return res + + COUNTER = Counter() + +One way to address this is to refactor the Counter class so that it +does less when constructed, delaying some work until first use. For +example: + +.. code-block:: python + + from non_strict import get_counter_start + + class Counter: + def __init__(self) -> None: + self.value: int = -1 + + def next(self) -> int: + if self.value == -1: + self.value = get_counter_start() + res = self.value + self.value += 1 + return res + COUNTER = Counter() + +Another approach is that instead of constructing the singleton at the +top of the file you can push this into a function so it gets defined +the first time it'll need to be used: + +.. code-block:: python + + _COUNTER = None + + def get_counter() -> Counter: + global _COUNTER + if _COUNTER is None: + _COUNTER = Counter() + return _COUNTER + +You can also use an lru_cache instead of a global variable: + +.. code-block:: python + + from functools import lru_cache + + @lru_cache(maxsize=1) + def get_counter() -> Counter: + return Counter() diff --git a/CinderDoc/strict_modules/guide/conversion/splitting_modules.rst b/CinderDoc/strict_modules/guide/conversion/splitting_modules.rst new file mode 100644 index 00000000000..265024c2aa4 --- /dev/null +++ b/CinderDoc/strict_modules/guide/conversion/splitting_modules.rst @@ -0,0 +1,69 @@ +Splitting Modules +################# + +Sometimes a module might contain functionality which is dependent upon certain +behavior which cannot be analyzed - either it truly has external side effects, +it is dependent upon another module which cannot yet be strictified and needs +to be used at the top-level, or it is dependent upon something which strict +modules have not yet been able to analyze. + +In these cases one possible solution, although generally a last resort, +is to break the module into two modules. The first module will only contain +the code which cannot be safely strictified. The second module will contain +all of the code that can be safely treated as strict. A better way to do this +is to not have the unverifable code happen at startup, but if that's not +possible then splitting is an acceptable option. + +Because strict modules can still import non-strict modules the strict module +can continue to expose the same interface as it previously did, and no other +code needs to be updated. The only limitation to this is that it requires +that the module being strictified doesn't need to interact with the non-strict +elements at the top level. For example classes could still create instances +of them, but the strict module couldn't call functions in the non-strict +module at the top level. + + +.. code-block:: python + + import csv + from random import choice + + FAMOUS_PEOPLE = list(csv.reader(open('famous_people.txt').readlines())) + + class FamousPerson: + def __init__(self, name, age, height): + self.name = name + self.age = int(age) + self.height = float(height) + + def get_random_person(): + return FamousPerson(*choice(FAMOUS_PEOPLE)) + + +We can split this into two modules, one which does the unverifable read of our +sample data from disk and another which returns the random piece of sample data: + + +.. code-block:: python + + import csv + + FAMOUS_PEOPLE = list(csv.reader(open('famous_people.txt').readlines())) + + +And we can have another module which exports our FamousPerson class along with +the API to return a random famous person: + +.. code-block:: python + + from random import choice + from famous_people_data import FAMOUS_PEOPLE + + class FamousPerson: + def __init__(self, name, age, height): + self.name = name + self.age = int(age) + self.height = float(height) + + def get_random_person(): + return FamousPerson(*choice(FAMOUS_PEOPLE)) diff --git a/CinderDoc/strict_modules/guide/conversion/stubs.rst b/CinderDoc/strict_modules/guide/conversion/stubs.rst new file mode 100644 index 00000000000..db3f047eba1 --- /dev/null +++ b/CinderDoc/strict_modules/guide/conversion/stubs.rst @@ -0,0 +1,67 @@ +Strict Module Stubs +################### + +Sometimes your modules depend on other modules that cannot be directly +strictified - it could depend on a Cython module, or a module from a +third-party library whose source code you can't modify. + +In this situation, if you are certain that the dependency is strict, you +can provide a strict module stub file (`.pys`) describing the behavior of +the module. Put the strict module stub file in your strict module stubs directory +(this is configured via `-X strict-module-stubs-path=...=` or +`PYTHONSTRICTMODULESTUBSPATH` env var, or by subclassing `StrictSourceFileLoader` +and passing a `stub_path` argument to `super().__init__(...)`.) + +There are two ways to stub a class or function in a strict module stub file. +You can provide a full Python implementation, which is useful in the case +of stubbing a Cython file, or you can just provide a function/class name, +with a `@implicit` decorator. In the latter case, the stub triggers the +strict module analyzer to look for the source code on `sys.path` and analyze +the source code. + +If the module you depend on is already actually strict-compliant you can +simplify the stub file down to just contain the single line `__implicit__`, +which just says "go use the real module contents, they're fine". +See `Lib/compiler/strict/stubs/_collections_abc.pys` for an existing example. +Per-class/function stubs are only needed where the stdlib module does +non-strict things at module level, so we need to extract just the bits we +depend on and verify them for strictness. + +If both a `.py` file and a `.pys` file exist, the strict module analyzer will +prioritize the `.pys` file. This means adding stubs to existing +modules in your codebase will shadow the actual implementation. +You should probably avoid doing this. + +Example of Cython stub: + +**myproject/worker.py** + +.. code-block:: python + + from some_cython_mod import plus1 + + two = plus1(1) + + +Here you can provide a stub for the Cython implementation of `plus1` + +**strict_modules/stubs/some_cython_mod.pys** + +.. code-block:: python + + # a full reimplementation of plus1 + def plus1(arg): + return arg + 1 + +Suppose you would like to use the standard library functions `functools.wraps`, +but the strict module analysis does not know of the library. You can add an implicit +stub: + +**strict_modules/stubs/functools.pys** + +.. code-block:: python + + @implicit + def wraps(): ... + +You can mix explicit and implicit stubs. See `Lib/compiler/strict/stubs` for some examples. diff --git a/CinderDoc/strict_modules/guide/conversion/testing.rst b/CinderDoc/strict_modules/guide/conversion/testing.rst new file mode 100644 index 00000000000..8007ff60acb --- /dev/null +++ b/CinderDoc/strict_modules/guide/conversion/testing.rst @@ -0,0 +1,32 @@ +Testing +####### + +You might be wondering how you're going to go and mock out functionality in +a strict module when we've already asserted that strict modules are immutable. +While we certainly don't want you to modify strict modules in production +they can be monkey patched in testing scenarios! + +To enable patching strict modules during testing, you will need to customize +your strict loader (see :doc:`../quickstart`) by creating a subclass and +passing `enable_patching = True` before installing the loader. + +.. code-block:: python + + from compiler.strict.loader import StrictSourceFileLoader + from typing import final + + @final + class StrictSourceFileLoaderWithPatching(StrictSourceFileLoader): + def __init__(self) -> None: + # ... + super().__init__( + # ... + enable_patching = True, + # ... + ) + +With patching enabled, you will be able to patch symbols in strict modules: + +.. code-block:: python + + mystrictmodule.patch("name", new_value) diff --git a/CinderDoc/strict_modules/guide/errors/class_attr_conflict.rst b/CinderDoc/strict_modules/guide/errors/class_attr_conflict.rst new file mode 100644 index 00000000000..464c6ffee50 --- /dev/null +++ b/CinderDoc/strict_modules/guide/errors/class_attr_conflict.rst @@ -0,0 +1,49 @@ +ClassAttributesConflictException +################################ + + Class member conflicts with instance member: foo + + +Strict modules require that instance attributes are distinct from class level +attributes such as methods. + +.. code-block:: python + + class C: + def __init__(self, flag: bool): + if flag: + self.f = lambda: 42 + + def f(self): + return '42' + + +``ClassAttributesConflictException`` 'Class member conflicts with instance member: f' + +In this example we are attempting to override a method defined on the class +with a unique per-instance method. We cannot do this in a strict module +because the instance attribute is actually defined at the class level. + + +.. code-block:: python + + class C: + value = None + def __init__(self, flag: bool): + if flag: + self.value = 42 + + + +``ClassAttributesConflictException`` 'Class member conflicts with instance member: value' + +In this example we're attempting to provide a fallback value that's declared +at the class level. + +In both of these cases the solution is to define the value either +completely at the instance or class level. For example we could change the +first example to always set `self.f` in the constructor, just sometimes setting +it to the default value. + +For additional information see the section on +:doc:`/guide/limitations/class_attrs`. diff --git a/CinderDoc/strict_modules/guide/errors/import_star_disallowed.rst b/CinderDoc/strict_modules/guide/errors/import_star_disallowed.rst new file mode 100644 index 00000000000..05397391fa3 --- /dev/null +++ b/CinderDoc/strict_modules/guide/errors/import_star_disallowed.rst @@ -0,0 +1,23 @@ +ImportStarDisallowedException +############################# + + Strict modules may not import ``*``. + +This error indicates that you are attempting to do a ``from module import *``. + +.. code-block:: python + + from foo import * + + +Strict modules simply outright prohibit this construct. Import stars are not +only generally considered bad style but they also make it impossible to +understand what attributes are defined within a module. Because import * can +bring in any name it has the possibility of overwriting existing names that +have been previously imported. + +To work around this explicitly import the values that you intend to use from +the module. + +For additional information see the section on +:doc:`/guide/limitations/imports`. diff --git a/CinderDoc/strict_modules/guide/errors/index.rst b/CinderDoc/strict_modules/guide/errors/index.rst new file mode 100644 index 00000000000..6510522f720 --- /dev/null +++ b/CinderDoc/strict_modules/guide/errors/index.rst @@ -0,0 +1,61 @@ +Intro to Errors +############### + +First, read the error and the documentation page linked from the error +message to understand what it's telling you! + +One common theme in all of these error messages will be the concept of an +"unknown value". An unknown value in a strict module is a value that comes +from a source that the strict module doesn't understand. That typically means +it's imported from a non-strict module or from an operation on an unknown +value. The error message will attempt to give you detailed information on the +source of the value, including the initial unknown name that was imported or +used and the chain of operations performed against that value. + +Typically an unknown value will be displayed as one of ```` where unknown is the name of the non-strict imported module, +```` where the value comes from ``from +unknown import some_name``, or just a simple name like ```` if the +value comes from an undefined global or unknown built-in. + +There are a few basic causes of errors: + +* Your code is actually doing something side-effecty or unsafe in top-level + code (including functions called from top-level code, e.g. decorators), and + strict modules is alerting you to the problem. In this case you should adjust + your code to not do this. :doc:`../conversion/singletons` has some advice on + moving side-effecty code out of the import codepath by making it lazy. + +* Your code is actually fine, but you are using at module level + some class or function that you imported from a non-strict module, so our + analysis doesn't know about it and is flagging that you are using an "unknown + value" at import time. Early in adoption, this will likely be a common case. + Options for dealing with it: + + a. You can try converting the module you are importing from to be strict + itself. If the module is hard to convert, but the specific piece of it you + need is not, you could :doc:`split the module + <../conversion/splitting_modules>`. + b. If the dependency is external to your codebase (i.e. third-party), + you can add a :doc:`stub file <../conversion/stubs>` to tell + strict modules about it. + c. If (a) and (b) are hard and you need to unblock yourself, you can + remove `import __strict__` from your module for now. This could require + de-strictifying other modules as well, if other strict modules import from + yours. + +* Your code is actually fine, but strict modules is not able to analyze it + correctly (e.g. a missing built-in, or some aspect of the language we aren't + fully analyzing correctly yet). In this case you should just remove + `import __strict__` to unblock yourself and report a bug so we + can fix the problem. + +For guidance on specific errors please refer to this list: + + +.. toctree:: + :maxdepth: 1 + :titlesonly: + :glob: + + * diff --git a/CinderDoc/strict_modules/guide/errors/modify_imported_value.rst b/CinderDoc/strict_modules/guide/errors/modify_imported_value.rst new file mode 100644 index 00000000000..a64aac47751 --- /dev/null +++ b/CinderDoc/strict_modules/guide/errors/modify_imported_value.rst @@ -0,0 +1,15 @@ +StrictModuleModifyImportedValueException +######################################## + + from module is modified by ; this is + prohibited. + +Strict modules only allow a module to modify values that are defined/created +within the defining module. This prevents one module from having side effects +that would impact another module or non-determinism based upon the order of +imports. + +For additional information see the section on +:doc:`/guide/limitations/ownership`. + +For guidance on how to fix this see :doc:`/guide/conversion/external_modification`. diff --git a/CinderDoc/strict_modules/guide/errors/prohibited_callable.rst b/CinderDoc/strict_modules/guide/errors/prohibited_callable.rst new file mode 100644 index 00000000000..576f58cfd56 --- /dev/null +++ b/CinderDoc/strict_modules/guide/errors/prohibited_callable.rst @@ -0,0 +1,25 @@ +ProhibitedBuiltinException +########################## + + Call to built-in '' is prohibited at module level. + +This error indicates that you are attempting to call a known built-in function +which strict modules do not support. + +Currently this error applies to calling exec and eval. + + +.. code-block:: python + + exec('x = 42') + + +Currently strict modules do not support exec or eval at the top level. If you +must use them you can move them into a function which lazily computes the value +and stores it in a global variable. See :doc:`../conversion/singletons` as +one possible solution to this. + +In the future strict modules may support exec/eval as long as the values being +passed to them are deterministic. This would enable more complex library code +to be defined within strict modules. One example of this in real-world code +is Python's namedtuple class which uses exec to define tuple instances. diff --git a/CinderDoc/strict_modules/guide/errors/unknown_call.rst b/CinderDoc/strict_modules/guide/errors/unknown_call.rst new file mode 100644 index 00000000000..898e20d63e6 --- /dev/null +++ b/CinderDoc/strict_modules/guide/errors/unknown_call.rst @@ -0,0 +1,65 @@ +UnknownValueCallException +######################### + + Module-level call of non-strict value 'function()' is prohibited. + +This error indicates that you are attempting to call a value at module level +which the strict modules analysis does not understand. + +Currently the most likely reason you would see this error is because you are +importing something from a non-strict module and then calling it at module +level. For example: + +.. code-block:: python + + import __strict__ + from nonstrict_module import something + + x = something() + + +This code will result in an error message such as: + +``UnknownValueCallException`` 'Call of unknown value 'something()' is prohibited.' + +If `something()` has no side effects (it only returns a value), your options, +in order of preference, are to a) strictify the `nonstrict_module`, b) call +`something()` lazily on demand rather than at module level, or c) +de-strictify the module you are currently working in. If `something()` does +in fact have side effects, the only option you should consider is (b). + +Another case where you might see this is with an unsupported builtin: + +.. code-block:: python + + import __strict__ + + print('hi') + + +In this case we're attempting to call a built-in function which strict +modules don't support. Printing is typically a side effect so we don't +currently support it at module level. There are a number of other built-ins +which aren't currently supported as well. For the full list of supported +builtins see :doc:`/guide/limitations/builtins`. + +We can fix this case by removing the usage of the built-in at the top-level. + +If the function is a built-in and something you think strict modules should +support at module level, you might want to report a bug! + +.. code-block:: python + + import __strict__ + + ALPHABETE = lsit('abcdefghijklmnopqrstuvwxyz') + + +``UnknownValueCallException`` 'Call of unknown value 'lsit()' is prohibited.' + +In this case we've simply made a mistake and misspelled the normal built-in +list. One nice thing is that strict modules will detect this and give you +an early warning. We can fix it by just fixing the spelling. + +You can look at :ref:`conversion_tips` for other possible solutions to issues +like this. diff --git a/CinderDoc/strict_modules/guide/errors/unknown_value_attribute.rst b/CinderDoc/strict_modules/guide/errors/unknown_value_attribute.rst new file mode 100644 index 00000000000..b01ab4221e4 --- /dev/null +++ b/CinderDoc/strict_modules/guide/errors/unknown_value_attribute.rst @@ -0,0 +1,41 @@ +UnknownValueAttributeException +############################## + + Module-level attribute access on non-strict value 'value.attr' is prohibited. + +This error indicates that you are attempting to access an attribute on a +value that can't be analyzed by strict modules (e.g. because it is imported +from a non-strict module). Because attribute access in Python can execute +arbitrary code, doing this on an unknown value (where we can't analyze the +effects) could cause arbitrary side effects and is prohibited at module +level. + +.. code-block:: python + + from nonstrict import something + + class MyClass(something.SomeClass): + pass + +This code will result in an error message such as: + +``UnknownValueAttributeException`` '.SomeClass' + +The error tells you both what unknown value you are accessing an attribute +on, and the name of the attribute. + +One possible solution to this is making the nonstrict module strict so that +``something`` can be used at the top-level. If the nonstrict module is not in +your codebase you could create a :doc:`stub file <../conversion/stubs>` for it. + +In a case like the above example where the unknown value is an imported module, +you can also solve it like this: + +.. code-block:: python + + from nonstrict.something import SomeClass + + class MyClass(SomeClass): + pass + +You can look at :ref:`conversion_tips` for more ways to fix this error. diff --git a/CinderDoc/strict_modules/guide/errors/unknown_value_binary_op.rst b/CinderDoc/strict_modules/guide/errors/unknown_value_binary_op.rst new file mode 100644 index 00000000000..7ca0aec0858 --- /dev/null +++ b/CinderDoc/strict_modules/guide/errors/unknown_value_binary_op.rst @@ -0,0 +1,28 @@ +UnknownValueBinaryOpException +############################# + + Module-level binary operation on non-strict value 'lvalue [op] rvalue' is prohibited. + +This error indicates that you are attempting to perform a binary operation on +a value that can't be analyzed by strict modules (e.g. because it is imported +from a non-strict module). Because binary operations can be overridden in +Python to execute arbitrary code, doing this on an unknown value can cause +arbitrary side effects and is prohibited at top level of a strict module. + +.. code-block:: python + + from nonstrict import SOME_CONST + + MY_CONST = SOME_CONST + 1 + +This code will result in an error message such as: + +``UnknownValueBinaryOpException`` ' + 1' + +The error tells you strict modules' understanding of the values on each side +of the binary operation, and what the operation itself is. + +Typically the best solution to this situation is to make the module +containing ``SOME_CONST`` strict. + +You can look at :ref:`conversion_tips` for more ways to fix this error. diff --git a/CinderDoc/strict_modules/guide/errors/unknown_value_bool_op.rst b/CinderDoc/strict_modules/guide/errors/unknown_value_bool_op.rst new file mode 100644 index 00000000000..52360a16040 --- /dev/null +++ b/CinderDoc/strict_modules/guide/errors/unknown_value_bool_op.rst @@ -0,0 +1,30 @@ +UnknownValueBoolException +######################### + + Module-level conversion to bool on non-strict value 'value' is prohibited. + +This error indicates that you are attempting to convert a value to a bool, +either explicitly by calling bool(value) on it or implicitly by using it +in a conditional location (e.g. if, while, if expression, or, and, etc...) + + +.. code-block:: python + + from nonstrict_module import x + + if x: + pass + +This code will result in an error message such as: + +``UnknownValueBoolException`` 'Conversion to bool on unknown +value '{x imported from nonstrict_module}' is prohibited.' + +Here you can see the error tells you the value which is being checked for +truthiness. + +One possible solution to this is making the nonstrict module strict so that +x can be used at the top-level. If the non-strict module is something from +the Python standard module you might want to report a bug! + +You can look at :ref:`conversion_tips` for more ways to fix this error. diff --git a/CinderDoc/strict_modules/guide/errors/unknown_value_index.rst b/CinderDoc/strict_modules/guide/errors/unknown_value_index.rst new file mode 100644 index 00000000000..7474cfab774 --- /dev/null +++ b/CinderDoc/strict_modules/guide/errors/unknown_value_index.rst @@ -0,0 +1,28 @@ +UnknownValueIndexException +########################## + + Module-level index into non-strict value 'value[index]' is prohibited. + +This error indicates that you are attempting to index into a value that +strict modules can't analyze. + +.. code-block:: python + + from nonstrict import GenericClass + + class MyClass(GenericClass[int]): + pass + +This code will result in an error message such as: + +``UnknownValueIndexException`` '[]' + +Here you can see the error tells you both what value we are accessing which +is not statically analyzable, but also what value we are using to +index into the unknown value (in this case the int type). + +One possible solution to this is making the nonstrict module strict so that +GenericClass can be used at the top-level. If the non-strict module is +something from the Python standard library you might want to report a bug! + +You can look at :ref:`conversion_tips` for more ways to fix this error. diff --git a/CinderDoc/strict_modules/guide/errors/unsafe_call.rst b/CinderDoc/strict_modules/guide/errors/unsafe_call.rst new file mode 100644 index 00000000000..e63a2af7949 --- /dev/null +++ b/CinderDoc/strict_modules/guide/errors/unsafe_call.rst @@ -0,0 +1,37 @@ +UnsafeCallException +################### + + Call 'function()' may have side effects and is prohibited at module level. + +This error indicates that you are attempting to call a function whose execution +will cause side effects or has elements which can not be successfully verified. + + +.. code-block:: python + + import __strict__ + + def side_effects(): + print("I have side effects") + return 42 + + FORTY_TWO = side_effects() + +This code will result in an error message such as: + +``UnsafeCallException`` 'Call 'side_effects()' may have side effects and +is prohibited at module level.' + +In addition to this error from the call site you'll see an error about the +underlying violation in the function. Here the underlying error turned out +to be a :doc:`unknown_call`. In this case strict modules has no +definition of print because its sole purpose is to cause side effects: + +``UnknownValueCallException`` Call of unknown value 'print()' is prohibited. + +You can consider removing the prohibited operation from the function, +or move the call to the side effecting function out of the top-level. +You can look at :ref:`conversion_tips` for more ways to fix this error. + +If the underlying operation in the function is something you think strict +modules should support the analysis of you might want to report a bug! diff --git a/CinderDoc/strict_modules/guide/index.rst b/CinderDoc/strict_modules/guide/index.rst new file mode 100644 index 00000000000..42a4d22870f --- /dev/null +++ b/CinderDoc/strict_modules/guide/index.rst @@ -0,0 +1,75 @@ +Users Guide +########### + +Getting Started +--------------- + +Writing your first strict module is super easy. Check out the +:doc:`quickstart` to find out more. + +What does it mean if my module is strict? +----------------------------------------- +The short version of the strict-module rules: your module may not do anything +dynamic at import time that might have external side effects; it should just +define a set of names (constants, functions, and classes). + +While that might seem a little bit limiting, there is still a wealth of +built-ins that are supported. You can still do a lot of reflection over +types, use normal constructs like lists and dictionaries, comprehensions, +and a large number of built-in functions within your module definition. + +Making your modules strict means you receive benefits of strictness, including +but not limited to increased reliability and performance improvements. + + +Limitations +----------- +Strict modules undergo a number of different checks to ensure that they will +reliably and consistently produce the same module for the same input source +code. This ensures that top-level modules will always reliably succeed and +won't take dependencies on external environmental factors. + +.. toctree:: + :maxdepth: 1 + :titlesonly: + :glob: + + limitations/* + + +.. _conversion_tips: + +Conversion Tips +--------------- +This section includes tips on how to convert a non-strict module into a strict +module, common problems which might come up, and how you can effectively +:doc:`test ` your strict modules even though they're +immutable. + + +.. toctree:: + :maxdepth: 1 + :titlesonly: + :glob: + + conversion/* + + +Strict Module Errors / Exceptions +--------------------------------- +When converting a module to strict, or when modifying an existing strict +module, you may see many different errors reported by the linter. These will +have an exception name associated with them along with a detailed message +explaining what and where something went wrong in verifying a module as +strict. This section has detailed description of what error means and how you +can possibly fix it. + + + +.. toctree:: + :maxdepth: 1 + :titlesonly: + :glob: + + errors/index + errors/* diff --git a/CinderDoc/strict_modules/guide/limitations/builtins.rst b/CinderDoc/strict_modules/guide/limitations/builtins.rst new file mode 100644 index 00000000000..8360044c314 --- /dev/null +++ b/CinderDoc/strict_modules/guide/limitations/builtins.rst @@ -0,0 +1,56 @@ +Supported Builtins +################## + +Strict modules support static verification for a subset of the standard Python +builtin types and functions. Usage of builtins not in this +list at the top-level of a module will result in an error message +and will prevent the module from being able to be marked as strict. + +If you need support for an additional built-in function or type outside of the +supported list please report a bug! + +Supported types and values: + +* AttributeError +* bool +* bytes +* classmethod +* complex +* dict +* object +* Ellipsis +* Exception +* float +* int +* list +* None +* NotImplemented +* property +* range +* set +* staticmethod +* str +* super +* type +* TypeError +* tuple +* ValueError + +Supported functions: + +* callable +* chr +* getattr +* hasattr +* len +* max +* min +* ord +* print +* isinstance +* setattr + +Unsupported functions: + +Currently exec and eval are disallowed at the top-level. In the future we may +allow their usage if they are used with deterministic strings. diff --git a/CinderDoc/strict_modules/guide/limitations/class_attrs.rst b/CinderDoc/strict_modules/guide/limitations/class_attrs.rst new file mode 100644 index 00000000000..2c5c5bbe485 --- /dev/null +++ b/CinderDoc/strict_modules/guide/limitations/class_attrs.rst @@ -0,0 +1,111 @@ +Class Attributes +################ + +Strict modules transform the way attributes are handled in Python from being +diffused between types and instances to be entirely declared on the type. + +For normal Python definitions when you define a class, the class will have a +dictionary, its sub-types will have dictionaries, and the instances will also +have dictionaries. When an attribute is looked up you typically have to look in +at least the class dictionary and the instance dictionary. In strict modules +we've removed the instance dictionary and replaced it with attributes that +are always defined at the class level. We've done this by leveraging a +standard Python feature - ``__slots__``. + +There are a few different benefits from this transformation. The first is that +slots generally provide faster access to an attribute than instance +dictionaries - to access an instance field first the type's dictionary needs +to be checked, and then the instance dictionary can be checked. With this +transformation in place we only ever need to look at the type's dictionary +and we're done. We've also applied additional optimizations within Cinder to +further improve the performance of this layout. + +Another benefit is that it uses less memory - with a fixed number of slots +Python knows exactly the size of the instance that it needs to allocate to +store all of its fields. With a dictionary it may quickly need to be resized +multiple times while allocating the object, and dictionaries have a load factor +where a percentage of slots are necessarily unused. The dictionary is also its +own object instead of storing the attributes directly on the instance. + +And a final benefit is developer productivity. When you can arbitrarily attach +any attribute to an instance it's easy to make a mistake where you have a typo +on a member name and don't understand why it's not being updated. By not +allowing arbitrary fields to be assigned we turn this into an immediate and +obvious error. + +Class Attributes in Detail +-------------------------- + +Now let's look look at some detailed examples. This first example shows what +you can do today in Python without a strict module to get the same benefits: + +.. code-block:: python + + class C: + __slots__ = ('myattr', ) + def __init__(self): + self.myattr = None + + a = C() + a.my_attr = 42 # AttributeError: 'C' object has no attribute 'my_attr' + +In this case we've defined a class using Python's ``__slots__`` feature and +specified that the type has an attribute "myattr". Later we've assigned +to "my_attr" accidentally. Without the presence of ``__slots__`` the Python +runtime would have happily allowed this assignment and we might have spent +a lot of time debugging while myattr doesn't have the right value. + +Strict modules handles this transformation to ``__slots__`` automatically and +will typically not require extra intervention on the behalf of the +programmer. If we put this code into a strict module all we have to do +is remove the ``__slots__`` entry and we get the exact same behavior: + +.. code-block:: python + + import __strict__ + + class C: + def __init__(self): + self.myattr = None + + a = C() + a.my_attr = 42 # AttributeError: 'C' object has no attribute 'my_attr' + +Strict modules will automatically populate the entries for ``__slots__`` based +upon the fields that are assigned in ``__init__``. If you have fields which you +don't want to eagerly populate you can also use Python's class level +annotations to indicate the presence of a field: + +.. code-block:: python + + class C: + myattr: int + + a = C() + a.myattr = 42 # OK + +We anticipate this shouldn't be much of an additional burden on developers +because these annotations are already used for providing typing information +to static analysis tools like Pyre. + +But these changes do have some subtle impacts - for example this code is +now an error where it wasn't before: + +.. code-block:: python + + class C: + def __init__(self): + self.f = 42 + + def f(self): + return 42 + + # Strict Module error: Class member conflicts with instance member: f + +The problem here is that there's now contention for storing two things in the +type - one is the method for "f", and the other is storing a descriptor (an +object which knows where to get or set the value in the instance) for the +instance attribute. It's not very often that users want to override a class +attribute with an instance one, but when it occurs you'll need to +resort to other techniques. For information on how to handle this see +:doc:`../conversion/class_inst_conflict`. diff --git a/CinderDoc/strict_modules/guide/limitations/deterministic.rst b/CinderDoc/strict_modules/guide/limitations/deterministic.rst new file mode 100644 index 00000000000..ec1b511a7d9 --- /dev/null +++ b/CinderDoc/strict_modules/guide/limitations/deterministic.rst @@ -0,0 +1,27 @@ +Deterministic Execution +####################### + +One of the most significant limitations, and the primary point of strict +modules, is to enforce that their contents are deterministic. This is +achieved by having an allow-list approach of what can occur inside of a strict +module at import time. The allow-list is composed of standard Python syntax +and the set of :doc:`builtins` which are allowed. Use of anything that +isn't analyzable will result in an error when importing a strict module. + +Strict modules themselves are verified to conform to the allow list by +an analysis done with an interpreter. The interpreter +will precisely simulate the execution of your code, analyzing loops, +flow control, exceptions and all other typical Python language elements +that are available. This means that you have available to you the full +breadth of the language. + +Strict modules will only validate code at the module +top-level - that includes elements such as top-level class declarations, +annotations on functions (unless `from __future__ import annotations` is +applied), etc. It will also analyze the result of calling any functions from +the top-level (which includes decorators on top-level definitions). + +The result is that while you are more limited in what you can +do within your module definitions, your actual functions aren't limited in +what they are allowed to do. Effectively a slightly more limited Python is +your meta-programming language for defining your Python programs. diff --git a/CinderDoc/strict_modules/guide/limitations/imports.rst b/CinderDoc/strict_modules/guide/limitations/imports.rst new file mode 100644 index 00000000000..112eb8eea98 --- /dev/null +++ b/CinderDoc/strict_modules/guide/limitations/imports.rst @@ -0,0 +1,86 @@ +Imports +####### + +Import Mechanics +================ + +There is also some impact on how imports behave. Because strict modules are +immutable, child packages cannot be set on them when the child package is +imported. Given that this is also a common source of errors and can cause +issues with order of imports this is a good thing. + +You might have some code which does a `from package import child` in one +spot, and elsewhere you might do `import package` and then try and access +`package.child`. When `package` is a strict module this will fail because +`child` is not published. + +Explicitly Imported Child Packages +---------------------------------- + +If you'd like to enable this programming model you can still explicitly +publish the child module on the parent package + +**package/__init__.py** + +.. code-block:: python + + import __strict__ + + from package import child + + +**package/child.py** + +.. code-block:: python + + import __strict__ + + def foo(): pass + +This pattern is also okay and the child package will be published on the +parent package. + + +Using Imports +============= + +You also may need to be a little bit careful about how imported values are +used within a strict module. In order to verify that a strict module has no +side effects you cannot interact with any values from non-strict modules at +the top-level of your module. While it may typically be obvious when you +are interacting with non-strict values there's at least one case when its +less obvious. + + +**package/__init__.py** + +.. code-block:: python + + + # I'm not strict + +**package/a.py** + +.. code-block:: python + + import __strict__ + + class C: + pass + +**strict_mod.py** + +.. code-block:: python + + import __strict__ + + from package import a + + class C(a.C): # not safe + pass + +In this case using "a" at the top-level is not safe because the package +itself isn't strict. Because the package isn't strict random code +could sneak in and replace "a" with a value which isn't the strict module. + +This can easily be solved by marking the package as strict. diff --git a/CinderDoc/strict_modules/guide/limitations/index.rst b/CinderDoc/strict_modules/guide/limitations/index.rst new file mode 100644 index 00000000000..fe4020424b6 --- /dev/null +++ b/CinderDoc/strict_modules/guide/limitations/index.rst @@ -0,0 +1,14 @@ +Limitations +########### + +Strict modules undergo a number of different checks to ensure that they will +reliably and consistently produce the same module for the same input source +code. This ensures that top-level modules will always reliably succeed and won’t +take dependencies on external environmental factors. + +.. toctree:: + :maxdepth: 1 + :titlesonly: + :glob: + + * diff --git a/CinderDoc/strict_modules/guide/limitations/ownership.rst b/CinderDoc/strict_modules/guide/limitations/ownership.rst new file mode 100644 index 00000000000..58c673cd2a0 --- /dev/null +++ b/CinderDoc/strict_modules/guide/limitations/ownership.rst @@ -0,0 +1,96 @@ +Ownership +######### + +Strict modules are analyzed with a concept of ownership. That is, every value +that is produced from strict modules is owned by one and only one strict +module, and only that strict module is capable of mutating it. This +requirement ensures that modules are deterministic, and there isn't state is +the system which develops in an ad-hoc manner based upon how different modules +are imported. + +For a value to be owned by a module it doesn't need to actually be created +directly within the module which owns it. Rather the owning module is the +caller that ultimately causes the value to be created. Consider for example +this code: + +**a.py** + +.. code-block:: python + + import __strict__ + + def f(): + return {} + +**b.py** + +.. code-block:: python + + import __strict__ + + from a import f + + + x = f() + x["name"] = "value" + +This example is fine and will be permitted by strict modules; the owner of +the dictionary referred to by the variable `x` is module `b`. + +There are other useful patterns of this sort of modification. For example a +decorator can safely be applied in a module which will mutate the defining +function: + +**methods.py** + +.. code-block:: python + + import __strict__ + + def POST(f): + f.method = 'POST' + return f + +**routes.py** + +.. code-block:: python + + import __strict__ + + from methods import POST + + @POST + def create_user(*args): + ... + + +But it bans other patterns which you may be used to. For example you cannot +use a decorator to create a registry of functions: + +**methods.py** + +.. code-block:: python + + import __strict__ + + ROUTES = set() + + def route(f): + ROUTES.add(f) + return f + +**routes.py** + +.. code-block:: python + + import __strict__ + + from methods import route + + @route + def create_user(*args): + ... + + +This will result in a ``StrictModuleModifyImportedValueException`` " +from module methods is modified by routes; this is prohibited." diff --git a/CinderDoc/strict_modules/guide/quickstart.rst b/CinderDoc/strict_modules/guide/quickstart.rst new file mode 100644 index 00000000000..1be0cae7f50 --- /dev/null +++ b/CinderDoc/strict_modules/guide/quickstart.rst @@ -0,0 +1,69 @@ +Quickstart +########## + +How do I use it? +---------------- + +Using Strict Modules requires a module loader able to detect strict modules +based on some marker (we use the presence of ``import __strict__``). +Such a loader is included (at `compiler.strict.loader.StrictSourceFileLoader`) +and you can install it by calling `compiler.strict.loader.install()` in the +"main" module of your program (before anything else is imported.) +Note this means the main module itself cannot be strict. Alternatively, set the +``PYTHONINSTALLSTRICTLOADER`` environment variable to a nonzero value, and +the loader will be installed for you (but then you can't customize the loader). + +How do I make my module strict? +------------------------------- + +To opt your module in, place the line ``import __strict__`` at the top of the +module. The ``__strict__`` marker line should come after the docstring if +present, after any ``from __future__ import`` statements, and before any +other imports. Comments can also precede the ``__strict__`` marker. + +If your module is marked as strict but violates the strict-mode rules, you +will get detailed errors when you try to import the module. + +What are the risks? +------------------- + +Most of the strict-mode restrictions have purely local effect; if you are +able to import your module after marking it strict, you're mostly good to go! +There are a couple runtime changes that can impact code outside the module: + +1. Strict mode makes the module itself and any classes in it immutable after +the module is done executing. This is most likely to impact tests that +monkeypatch the module or its classes. Refer to the :doc:`conversion/testing` +section to learn how to enable patching of strict moduels for testing. + +2. Instances of strict classes have `__slots__ +`_ automatically +created for them. This means they will raise ``AttributeError`` if you try to +add any attribute to them that isn't declared with a type annotation on the +class itself (e.g. ``attrname: int``) or assigned in the ``__init__`` method. +If you aren't confident that this isn't happening to your class somewhere in +the codebase, you can temporarily place the ``strict_modules.loose_slots`` +decorator on the class for a safer transition. See +:doc:`conversion/loose_slots` for details. + +What are the benefits? +---------------------- + +When you convert your module to strict, you immediately get these benefits: + +1. It becomes impossible to accidentally introduce import side effects in +your module, which prevents problems that can eat up debugging time or even +break prod. + +2. It becomes impossible to accidentally modify global state by mutating your +module or one of the classes in your module, also preventing bugs and test +flakiness. + +In the future, we hope that you will get other benefits too, like faster +imports when the module is unchanged since last import and production +efficiency improvements as well. + +What if I get a StrictModuleException? +-------------------------------------- + +See :doc:`errors/index` for advice on handling errors in your strict module. diff --git a/CinderDoc/strict_modules/index.rst b/CinderDoc/strict_modules/index.rst new file mode 100644 index 00000000000..24b709e9b9b --- /dev/null +++ b/CinderDoc/strict_modules/index.rst @@ -0,0 +1,40 @@ +Strict Modules +############## + + `Python is more dynamic than the language I *wanted* to design. -- Guido van Rossum `_ + + +What are strict modules? +------------------------ + +Strict modules are an opt-in mechanism for restricting the dynamism of +top-level (module-level) code in a Python module. These stricter +semantics are designed to have multiple benefits: + +* Eliminate common classes of developer errors +* Improve the developer experience +* Unlock new opportunities for optimizing Python code and simplify other classes of optimizations + +While strict modules alter what you can do at the top-level of your code +they don't limit what you can do inside of your function definitions. They +also are designed to not limit the expressiveness of what you can do even +at the top-level. The limits on strict modules at the top-level all have a single +goal: Make sure that module definitions can reliably be statically analyzed. + +In order to support this there are also some runtime changes for strict +modules, mostly around immutability. In order to be able to make guarantees +about the analysis we must be certain that it won't later be invalidated by +runtime changes. To this end strict modules themselves are immutable and +types defined within strict modules are immutable as well. + +.. toctree:: + :caption: Overview + :maxdepth: 2 + + guide/quickstart + +.. toctree:: + :caption: Users Guide + :maxdepth: 3 + + guide/index