Extending PyCharm auto-complete (generated run-time code)

Answered

I’m developing a small python package that uses auto-generated code in way that resembles python’s ctypes package usage.

 

Example code:

>>> class Point(Structure):

...     _fields_ = [Field("name", str),

...                 Field("x", (int, long)),

...                 Field("y", (int, long))]

... p = Point("p1", 10, 20)

... assert p.name.startswith("p") and p.x == 10

 

I wish to write a plugin that allows the IDE to understand these structures and provide:

* Auto-complete of the generated members (for the class / instances) – i.e. name, x and y

* Auto-complete for the generated-member’s members themselves (by resolving each member’s type) – i.e. startswith for name as a string

* Go-To-Def functionality for those members to their field descriptors.

* Hide the "unresolved attribute references..."

* Constructor’s parameters tool-tip.

 

I started addressing the first item on my list (successfully using CompletionContributor) and I think that it might cause a slight (unwanted) delay opening the auto-complete session.

I am wondering whether I’m going this the wrong way, and instead of handling each issue alone; I might want to allow PyCharm to recognize these structures while parsing the code (and generating "stub-fields" there) thus allowing the already provided capabilities of the IDE to do their magic. (I know that this doesn’t address the last issue, which will have to be dealt with separately…)

 

What is the correct way to address this feature?

Thanks,

Tal

14 comments
Comment actions Permalink

The simplest way for supporting all these code insight features for your class would be implementing the PyClassMembersProvider interface (in your example for providing name, x, y as the members of Point). You can use PyCustomMember and its typeCallback argument to provide the types of your class attributes. As a resolve target for attributes (e.g. for p.name) you can use the corresponding PyCallExpression for this field (e.g. Field("name", str)). Then you can use it in the type callback to infer the type 'str' for this attribute (see PyPsiFacade.parseTypeAnnotation('str') or TypeEvalContext.getType(PyReferenceExpression for the str in the Field(...) call).

1
Comment actions Permalink

Thanks! worked like a charm :)

One last thing, what about augmenting the constructor or extra functions? (and supplying a set of parameters)

0
Comment actions Permalink

You can implement a PyTypeProvider and supply your customized function parameters and their types.

0
Comment actions Permalink

Hey,

I'm having a small difficulty, I've implemented a PyTypeProvider for my construct function, and I used getCallType to return it. The inspection and auto-complete weren't using it..
I figured that they are calling and using getCallableType instead (and they are).

The problem is that I need to get the PyCallExpression in order to extract the class which is being invoked. Without it (when passing along only the PyCallable in the getCallableType) the class constructor's parameters cannot be inferred.

Is there a way around it? Maybe supplying a stub init function using a different provider mechanism? (such that findInitOrNew will return a special PyFunction that maintains the original class that was called upon)

Thanks,
Tal

0
Comment actions Permalink

Sorry for the late reply.

PyCharm should call PyTypeProvider.getCallType() for a call expression (PyCallExpression). But before that it adds a point of indirection by asking for the type of a callable object. See PyCallExpressionHelper.getCallType(), lines:

 

final PyType type = context.getType(callee);
if (type instanceof PyCallableType) {
final PyCallableType callableType = (PyCallableType)type;
return callableType.getCallType(context, call);
}

The best way to debug this problem is to create a unit test similar to tests in PyTypeTest.

0
Comment actions Permalink

Hey Andrey,

Your help is very much appreciated(!)

I think that when getCallableType is invoked in the context of creating the parameter info, there is no flow in which PyCallExpressionHelper.getCallType is called...


The callstack when getCallableType is invoked (for the parameter info as I understand):
1. PyArgumentList PyParameterInfoHandler.findElementForParameterInfo(@NotNull final CreateParameterInfoContext context)
2. PyArgumentsMapping PyCallExpressionImpl.mapArguments(@NotNull PyResolveContext resolveContext)
3. static PyCallExpression.PyArgumentsMapping PyCallExpressionHelper.mapArguments(@NotNull PyCallExpression callExpression, @NotNull PyResolveContext resolveContext, int implicitOffset)
    The next function on the stack is called with just the callable:
4. static List<PyParameter> PyUtil.getParameters(@NotNull PyCallable callable, @NotNull TypeEvalContext context)
5. static List<List<PyParameter>> PyUtil.getOverloadedParametersSet(@NotNull PyCallable callable, @NotNull TypeEvalContext context)
6. PyType TypeEvalContext.getType(@NotNull final PyTypedElement element)
7. PyType PyFunctionImpl.getType(@NotNull TypeEvalContext context, @NotNull TypeEvalContext.Key key)
8. PyType MyPyClassMembersProvider.getCallableType(@NotNull PyCallable callable, @NotNull TypeEvalContext context)

    When under 3 (PyCallExpressionHelper.mapArguments), the markedCalle was resolved under:
    4. PyMarkedCallee PyCallExpressionImpl.resolveCallee(PyResolveContext resolveContext, int offset)
    5. static PyCallExpression.PyMarkedCallee PyCallExpressionHelper.resolveCallee(PyCallExpression us, PyResolveContext resolveContext, int implicitOffset)
    6. PyFunction PyClassImpl.findInitOrNew(boolean inherited, final @Nullable TypeEvalContext context)
        ...
        which in turn returns the __init__ of the parent class,
        (not checking whether there is a different __init__ function provided)


In the cast of which there was a call to PyCallExpressionHelper.getCallType, I think it would not solve the problem because my callee is a PyReferenceExpression to a defined class ('Point' in the next example code), which goes to the first flow (commented "normal cases") and in turn gets resolved under getCallTargetReturnType (again going in to findInitOrNew).

>>> class Point(Structure):
...         _fields_ = [Field("name", str),
...                          Field("x", (int, long)),
...                          Field("y", (int, long))]
...  p = Point(


Attached here: http://pastebin.com/TGsHvYap is a short test case I wrote, if it is not too much trouble, may I ask what could I change in this type provider in order to supply a parameter info for a stub constructor for classes which do not declare one - having the name of the class as the name of the parameter?
Such that for the next code:
class A(object):
    pass
class B(A):
    pass
the parameter info for B() would have a parameter named 'B'?

Thanks again,
Tal

0
Comment actions Permalink

Sorry for the late reply. Is this problem still relevant to you?

0
Comment actions Permalink

yes please :)

0
Comment actions Permalink

Hey Tal,

 

Yes, `PyClass.findInitOrNew()` is bound to the PSI structure of the class and it doesn't allow customizing it via type providers. We have a somewhat similar problem with `collections.namedtuple`. You can follow this issue and get back to implementing parameter info for your `Structure` classes when it's fixed.

0
Comment actions Permalink

OK, will do - again, thank you.

0
Comment actions Permalink

@Tal Saiag, did you complete your plugin ? would it be a workaround for #PY-18246 ? If so would care to share it ? Bests, François.

0
Comment actions Permalink

Most of it yes, I thought about it as well :)

I just need to finish the merging process and testing the code + constructor (I saw the update and now working on a few merging issues with the current master - it's been a while since I pulled the updates)

I don't know why, but this stopped working:

PyPsiFacade.parseTypeAnnotation('str') or TypeEvalContext.getType(PyReferenceExpression for str)...

Both return null while testing the plugin under the 2017.1 EAP and when compiling PyCharm straight from the master after the tag PyCharm/171.3153.

Is there something new that I'm not aware of? or is it something from my end?

Thanks,

Tal

0
Comment actions Permalink

@Francois it looks like there are references in the code for typing.NamedTuple (with recent changes in the master and it seems they updated the issue fix version to 2017.1)..

Regarding overwriting the constructor - I haven't managed it so far myself..  not sure if it is possible in the current implementation when creating only a Plugin...

0
Comment actions Permalink

Did this plugin ever get released?

I would like to use it.

0

Please sign in to leave a comment.