Fixtures as describe arguments

[ROpdebee]

This has been requested and rejected before (#9, #11, #34) but I've done some investigation and I'd like to share my findings, and reopen the discussion.

Problem setting

Consider this (seemingly simple) scenario (shamelessly pulled from #9):

import pytest


def describe_books():
    @pytest.fixture
    def user():
        return ...

    @pytest.fixture
    def valid_book():
        return ...

    @pytest.fixture
    def invalid_book():
        return ...

    def describe_create_book(user):

        def with_valid_book(valid_book):
            # use user + valid_book fixtures ...

        def with_invalid_book(invalid_book):
            # use user + invalid_book fixtures ...

Seems simple enough, we want to use the user fixture in both with_valid_book and with_invalid_book, so why not add it as a funcarg fixture on the describe_create_book block instead of repeating ourselves? Since eliminating verbosity and repetitions is one of the goals of this plugin, it would make sense for this to be possible, and indeed, as evidenced by the linked issues, people assume this to be the case (me included when I first started using this plugin). However, it isn't, and such usage fails with a type error because of missing arguments.

Let's not focus on the less-than-descriptive error message, since a fix for that would be easy enough. Let's instead focus on whether injecting fixtures as describe block funcargs would be a possibility, and that writing

def describe_foo(fixt):
   def it_barks(other_fixt):
       ...

should be functionally equivalent to writing

def describe_foo():
    def it_barks(fixt, other_fixt):
       ...

Challenges

From previous issues, two main problems have popped up:

1. Describe blocks are evaluated during test collection, and have no access to fixtures.
2. New fixture values need to somehow be injected into the test function, i.e. the describe block's locals need to be adjusted before each test is executed.

(Potential) solutions

I'd argue that problem 1 isn't a big deal, considering that code inside the describe block itself isn't actually inside of a test, and thus probably shouldn't make use of the fixtures. Instead, some sort of dummy values could be given as arguments when the describe_ function is executed, assuming (and properly documenting) that these values should not be used outside of the tests. Perhaps some meta-magic with sys.settrace or sys.setprofile could be done to remove those arguments from the outer scope and raise some error when they're accessed.

Problem 2 is somewhat bigger, since we'd need to be able to somehow update the arguments given to the describe block before executing each test within the describe block with the value of the fixture. A straightforward way would be to just rerun the describe block with the new values, but that executes all of the contained code again, which might lead to severe problems. Luckily, embedded functions carry a __closure__ attribute, which is a tuple of "cells" that carry a reference to the locals of the parent function, i.e., the describe block. Since 3.7, the contents of these cells can be mutated. To exemplify:

def describe_block(foo):
    def inner(bar):
        return foo + bar
    return inner

inner = describe_block(2)
inner(2)  # 4
inner.__closure__[0].cell_contents = 4
inner(2)  # 6

Meaning that it could be possible to actually update these values without rerunning the describe block, thus solving problem 2, and perhaps making it possible to automatically inject the fixtures as funcargs.

Notes

• This wouldn't be possible in Python 3.6, since cell_contents is a read-only attribute and has only been made writeable since 3.7. I've experimented a bit, but it seems like they really didn't want you to do these sorts of things back then, as there's seemingly no way to instantiate those cell objects from within Python code either, but I may be missing something.
• Although the cell type has only been reified since 3.8 (https://docs.python.org/3.8/library/types.html#types.CellType), changing cell_contents is possible since 3.7
• I've successfully tested the cell_contents example above on 3.7.9, 3.8.2, and 3.9.0. I'm assuming it works on 3.10 as well, although I haven't checked.
• Am I sure this will work? No, but I believe this is the most promising way to get this to work. We'll only know for sure once there's actual code that does this, or fails to do it. I'm hoping to find some time in the near future to submit a WIP PR as a proof-of-concept.
• Is this some mad meta hacking? kinda sorta. But the discovery of local test functions is as well 🙂

[ropez]

Interesting. This is pretty much what I concluded was impossible in #11 (5 years ago). Apparently, it's now possible.

[Cito]

Unfortunately, I don't have the time to engage in this, but I agree it sounds interesting and if anyone wants to tackle it, please go ahead and create a PR.

[nyoma-diamond]

+1 to this for visibility since there hasn't been any discussion on this in nearly 4 years. Would definitely be a useful feature. Has the state of affairs changed at all to make this more feasible?