PEP 612 clarification: ParamSpec that captures a method signature

[erictraut]

I'm a contributor to pyright, and I'm hoping to get some clarification about a particular use case involving ParamSpec (introduced in PEP 612) and __call__ methods.

Consider the following code, which executes with no runtime exceptions.

from functools import cache

class Foo:
    def not_cached(self) -> int:
        return 0

    @cache
    def cached(self) -> int:
        return 0

foo = Foo()
foo.not_cached()
foo.cached()

The functools.cache decorator is defined in typeshed's functools.pyi as follows:

def cache(__user_function: Callable[_P, _T]) -> _lru_cache_wrapper[_P, _T]: ...

And _lru_cache_wrapper is defined as:

class _lru_cache_wrapper(Generic[_P, _T]):
    def __call__(self, *args: _P.args, **kwargs: _P.kwargs) -> _T: ...
    ...

When the @cache decorator is applied to the method cached, the ParamSpec _P captures the input parameters of cached. In other words, it captures the parameter list (self). The type of parameter self in this case is implied to be Foo or some subclass thereof (or if we adopt the terminology in draft PEP 637, its type is Self or Self@Foo).

The decorated type of cached is now _lru_cache_wrapper[(self: Self@Foo), int]. The __call__ method for this specialized class is effectively now typed as:

   def __call__(self: Self@_lru_cache_wrapper, self: Self@Foo) -> int: ...

For the call expression foo.cached() in the sample above, it appears that there is a missing argument (the one corresponding to the second Self@Foo parameter). Pyright therefore generates an error in this case, but clearly this isn't an error at runtime.

How do we avoid a false positive error in this case?

Have the authors of PEP 612 have considered this situation? If so, what is the solution? I haven't found anything in the PEP that describes the intended behavior.

Perhaps this is just the consequence of the functools.cache stub definition, which is attempting to model the runtime behavior but is in effect "lying" about some of the details? In that case, is there a better way to define functools.cache and similar decorators? Maybe through a use of descriptor?

[Akuli]

I think _lru_cache_wrapper should be a descriptor, not just a callable, because the runtime uses a def statement to create it.

[pradeep90]

@erictraut Mark isn't actively working on Pyre anymore, so I can step in for PEP 612 related questions.

The decorated type of cached is now _lru_cache_wrapper[(self: Self@Foo), int].

This implies that we need to leave out the self in the _lru_cache_wrapper.__call__ method since it is already provided by _P.args and _P.kwargs:

class _lru_cache_wrapper(Generic[_P, _T]):
    def __call__(*args: _P.args, **kwargs: _P.kwargs) -> _T: ...

The following code works in Pyre (you'd have to change pyre_extensions to typing, ParameterSpecification to ParamSpec):

from typing import *
from pyre_extensions import ParameterSpecification
    
_P = ParameterSpecification("_P")
_T = TypeVar("_T")
    
class _lru_cache_wrapper(Generic[_P, _T]):
    def __call__(*args: _P.args, **kwargs: _P.kwargs) -> _T: ...

def cache(__user_function: Callable[_P, _T]) -> _lru_cache_wrapper[_P, _T]: ...

class Foo:
    def not_cached(self) -> int:
        return 0

    @cache
    def cached(self, x: str, y: int) -> int:
        return 0

foo = Foo()
foo.not_cached()

foo.cached("hello", 42)

# Missing argument [20]: Call `_lru_cache_wrapper.__call__` expects argument `x`.
foo.cached()
# Incompatible parameter type [6]: In call `_lru_cache_wrapper.__call__`, for 1st positional only parameter expected `str` but got `int`.
foo.cached(1, 2)

If that sounds reasonable, we can probably update the PEP (along with a couple of clarifications that others had brought up).

[DevilXD]

There is a similar issue that it appears involves the same problem, in this PR: python/typeshed#6670

The thing is, removing self from _Wrapped.__call__ doesn't fix it. I'm not sure why, trying to follow the same logic as here tells me that it should, yet the error stays the same (at least according to the latest MyPy). I don't know how to debug MyPy so I'm unable to go any further with it.

It seems like it'd be cool to have some kind of a descriptor type, that'd handle inserting the self argument for methods, sort of like the existing classes use descriptors to generate bound methods.