Uncached Fixtures?

[ax3l]

Is there a way to deactivate caching in a fixture in pytest? Or is there a recommended way to write generators of objects that are uncached in pytest but still use decorators like parametrize?

This is based a somewhat on the original question on behavior of caches in #5642 (comment) and on how to use generators across tests that should yield objects that shall not be cached.

[ax3l]

The work-around I came up with for now, to keep all using the nice parametrizations in pytest, is to make the fixture return a generator (which can be cached w/o problems for my use case).

Original:

@pytest.fixture(scope="function", params=list(itertools.product([1, 3], [0, 1])))
def obj(C, D, request):
    A = request.param[0]
    B = request.param[1]
    obj = mymodule.Obj(A, B, C, D)
    return obj  # don't want to cache this between tests

Uncached fixture:

@pytest.fixture(scope="function", params=list(itertools.product([1, 3], [0, 1])))
def make_obj(C, D, request):

    def create():
        A = request.param[0]
        B = request.param[1]
        obj = mymodule.Obj(A, B, C, D)
        return obj

    return create

(in the test, the first thing I call is obj = make_obj().

[RonnyPfannschmidt]

It's absolutely unclear what you mean

Fixtures in function scope are not cached

And if you want to rely on __del__ for cleanup, ideally don't, it's absolutely broken

[ax3l]

It's absolutely unclear what you mean

Did you have a chance to see the example in the link in the description?
#5642 (comment)

# conftest.py
import gc
import sys
import numpy as np

import pytest


@pytest.fixture(scope="function")  # can I disable pytest caching for this yield?
def obj():
    v = np.array([1, 2, 3])
    print("startup: ", sys.getrefcount(v) - 1)  # refs: 1
    yield v
    gc.collect()
    print("\nteardown: ", sys.getrefcount(v) - 1)  # refs: 3 - why?
    del v  # still refs remaining

# test_foo.py
import sys

def test_foo(obj):
    print("++ test_foo ++")
    print("in test:", sys.getrefcount(obj) - 1)

Fixtures in function scope are not cached

Oh really? That's confusing to me, because in the linked example there are still 3 references (in CPython) on the function-scope yielded object:

startup:  1
++ test_foo ++
in test: 6
PASSED
teardown:  3

And if you want to rely on __del__ for cleanup, ideally don't, it's absolutely broken

Can you please be a bit more detailed what you mean by that?

[RonnyPfannschmidt]

the reference count cmoes from the fact that

pytest/src/_pytest/fixtures.py

Lines 1023 to 1044 in 3dac833

	def finish(self, request: SubRequest) -> None:
	exc = None
	try:
	while self._finalizers:
	try:
	func = self._finalizers.pop()
	func()
	except BaseException as e:
	# XXX Only first exception will be seen by user,
	# ideally all should be reported.
	if exc is None:
	exc = e
	if exc:
	raise exc
	finally:
	ihook = request.node.ihook
	ihook.pytest_fixture_post_finalizer(fixturedef=self, request=request)
	# Even if finalization fails, we invalidate the cached fixture
	# value and remove all finalizers because they may be bound methods
	# which will keep instances alive.
	self.cached_result = None
	self._finalizers = []

manages one of them an remvoes it after teardown,
the other reference comes from

pytest/src/_pytest/fixtures.py

Lines 544 to 549 in 3dac833

	def _fillfixtures(self) -> None:
	item = self._pyfuncitem
	fixturenames = getattr(item, "fixturenames", self.fixturenames)
	for argname in fixturenames:
	if argname not in item.funcargs:
	item.funcargs[argname] = self.getfixturevalue(argname)

preparing the parameters to pass over

storing references to the objects in the fixture lifecycle is a necessity to pass over the correct values at the correct point in time

a starting point for __del__ is https://docs.python.org/3/reference/datamodel.html#object.__del__

in practical terms - its demonstrably wrong to try to rly on __del__ for resource cleanup, its at best used for memory cleanup, and cant by design support clean resource cleanup

clean resource cleanup is best done with context managers

[stevenmanton]

Thanks for this discussion! The issue of the reference count is very sneaky and can cause some very subtle issues, particularly for people trying to control garbage collection. For example, this comes up often when testing multiple models that run on a GPU. This simple strategy does not work:

@pytest.fixture(scope="class")
def model():
    yield load_model()
    gc.collect()
    torch.cuda.empty_cache()

At first glance, one would expect the garbage collection to remove all references to the model and thus allow torch to free up the GPU memory. However, as @RonnyPfannschmidt pointed out, there are still references to the model! This means that if you run two GPU models back-to-back, Python might not have garbage-collected (and thus allowed torch to clear the GPU memory) by the time the second model loads. (See, for example, this discussion).

There seem to be two approaches that do work. First, you can garbage-collect before loading each model. This deletes all references to tensors on the GPU and that allows torch to reclaim that memory. For example:

# Approach 1:
@pytest.fixture(scope="class")
def model():
    gc.collect()
    yield load_model()

The second approach would be to delete all the tensors (or objects containing tensors) from the model and then garbage-collect. For example:

# Approach 2:
@pytest.fixture(scope="class")
def model():
    model = load_model()
    yield model
    del model.text_model
    del model.vision_model
    gc.collect()

Approach 2 has the advantage of clearing the memory after each test, though it's more boilerplate and it depends on each model.

[RonnyPfannschmidt]

Interesting find, this ought to become a contextmanager ideally dupplied bz torch