Exception on completion from compiled object using JPype #1351

Thrameos · 2019-06-22T23:15:23Z

This is a followup to #1347

Although completion is working for some cases, others are generating a RuntimeError in jedi.

Example:

s=jpype.java.lang.String("foo")
src='s.substring(1).con'
script = jedi.Interpreter(src,[locals()])
compl = [i.name for i in script.completions()]
print(compl)

Gives ['concat', 'contains', 'contentEquals']

Remove the "con" and we get...

s=jpype.java.lang.String("foo")
src='s.substring(1).'
script = jedi.Interpreter(src,[locals()])
compl = [i.name for i in script.completions()]
print(compl)

Gives

  """)
Traceback (most recent call last):
  File "/home/osboxes/python3-env/lib/python3.7/site-packages/jedi-0.14.0-py3.7.egg/jedi/api/completion.py", line 44, in filter_names
    k = (new.name, new.complete)  # key
  File "/home/osboxes/python3-env/lib/python3.7/site-packages/jedi-0.14.0-py3.7.egg/jedi/api/classes.py", line 439, in complete
    return self._complete(True)
  File "/home/osboxes/python3-env/lib/python3.7/site-packages/jedi-0.14.0-py3.7.egg/jedi/api/classes.py", line 408, in _complete
    if self._name.api_type == 'param' and self._stack is not None:
  File "/home/osboxes/python3-env/lib/python3.7/site-packages/jedi-0.14.0-py3.7.egg/jedi/evaluate/compiled/context.py", line 284, in api_type
    return next(iter(self.infer())).api_type
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "testJedi.py", line 18, in <module>
    compl = [i.name for i in script.completions()]
  File "/home/osboxes/python3-env/lib/python3.7/site-packages/jedi-0.14.0-py3.7.egg/jedi/api/__init__.py", line 216, in completions
    return completion.completions()
  File "/home/osboxes/python3-env/lib/python3.7/site-packages/jedi-0.14.0-py3.7.egg/jedi/api/completion.py", line 104, in completions
    return sorted(completions, key=lambda x: (x.name.startswith('__'),
RuntimeError: generator raised StopIteration

The text was updated successfully, but these errors were encountered:

davidhalter · 2019-06-23T00:02:14Z

I would need more information about why this is happening. Maybe use pdb to step through this error. I feel like the StopIteration shouldn't be coming from self.infer() being empty, because that's a ContextSet that's always containing one item.

Thrameos · 2019-06-23T00:12:30Z

Okay I will try to get some more information and get back to you.

Thrameos · 2019-06-23T02:29:22Z

I have isolated the problem somewhat. The problem occurs only if the completion matches a PyJPField type which is another opaque object. But this type is property like rather than function like. It doesn't have annotations or anything special that should be causing a problem. I have generated a trace of the two points with structure dumps. Unfortunately the traces are each 10M.

I will try to narrow it down further.

Thrameos · 2019-06-23T04:08:58Z

The key point of difference seems to be when it is collection the final list.

When successful it produces a CompiledName which results in a yield

jedi.evaluate.context.instance::infer(336):         for result_context in self._class_member_name.infer():
 --- modulename: jedi.cache, funcname: wrapper
{self:<class 'jedi.evaluate.compiled.context.CompiledName'>, func:<class 'function'>, name:_infer
}
jedi.cache::wrapper(45):         try:
jedi.cache::wrapper(46):             return getattr(self, name)
jedi.cache::wrapper(47):         except AttributeError:
jedi.cache::wrapper(48):             result = func(self)
 --- modulename: jedi.evaluate.compiled.context, funcname: infer
{self:<class 'jedi.evaluate.compiled.context.CompiledName'>
}
jedi.evaluate.compiled.context::infer(288):         return ContextSet([_create_from_name(
jedi.evaluate.compiled.context::infer(289):             self._evaluator, self.parent_context, self.string_name
 --- modulename: jedi.evaluate.compiled.context, funcname: _create_from_name
{evaluator:<class 'jedi.evaluate.Evaluator'>, compiled_object:<class 'jedi.evaluate.compiled.context.CompiledObject'>, name:substring
}
...
(LOTS more trace)
...
jedi.common.context::__iter__(46):         for element in self._set:
jedi.common.context::__iter__(46):         for element in self._set:
jedi.evaluate.context.instance::infer(336):         for result_context in self._class_member_name.infer():
jedi.common.context::__init__(17):         for context in iterable:
jedi.common.context::__init__(17):         for context in iterable:
jedi.evaluate.base_context::wrapper(434):         return ContextSet(func(*args, **kwargs))
 --- modulename: jedi.common.context, funcname: __iter__
{self:<class 'jedi.evaluate.base_context.ContextSet'>
}
jedi.common.context::__iter__(46):         for element in self._set:
jedi.common.context::__iter__(47):             yield element
jedi.common.context::__iter__(47):             yield element
 --- modulename: jedi.common.context, funcname: __iter__
{self:<class 'jedi.evaluate.base_context.ContextSet'>, element:<class 'jedi.evaluate.context.instance.CompiledBoundMethod'>
}
jedi.common.context::__iter__(47):             yield element

When failing it made a EmptyCompiledName, then fell quickly into an empty list.

jedi.evaluate.context.instance::infer(336):         for result_context in self._class_member_name.infer():
 --- modulename: jedi.evaluate.compiled.context, funcname: infer
{self:<class 'jedi.evaluate.compiled.context.EmptyCompiledName'>
}
jedi.evaluate.compiled.context::infer(369):         return NO_CONTEXTS
jedi.evaluate.compiled.context::infer(369):         return NO_CONTEXTS
 --- modulename: jedi.common.context, funcname: __iter__
{self:<class 'jedi.evaluate.base_context.ContextSet'>
}
jedi.common.context::__iter__(46):         for element in self._set:
jedi.common.context::__iter__(46):         for element in self._set:
jedi.evaluate.context.instance::infer(336):         for result_context in self._class_member_name.infer():
jedi.common.context::__init__(17):         for context in iterable:
jedi.common.context::__init__(17):         for context in iterable:
jedi.evaluate.base_context::wrapper(434):         return ContextSet(func(*args, **kwargs))
 --- modulename: jedi.common.context, funcname: __iter__
{self:<class 'jedi.evaluate.base_context.ContextSet'>
}
jedi.common.context::__iter__(46):         for element in self._set:
jedi.common.context::__iter__(46):         for element in self._set:
jedi.evaluate.compiled.context::api_type(284):         return next(iter(self.infer())).api_type
jedi.api.classes::_complete(408):         if self._name.api_type == 'param' and self._stack is not None:
jedi.api.classes::complete(439):         return self._complete(True)
jedi.api.completion::filter_names(44):             k = (new.name, new.complete)  # key
jedi.api.completion::completions(104):         return sorted(completions, key=lambda x: (x.name.startswith('__'),
 --- modulename: contextlib, funcname: __exit__

Does that help? I am applying a custom tracer with structure dumps on each call. I can look deeper if you want to isolate a particular structure,

Thrameos · 2019-06-23T04:24:14Z

Here is a better trace of the final moments.

jedi.evaluate.context.instance::infer(336):         for result_context in self._class_member_name.infer():
 --- modulename: jedi.evaluate.compiled.context, funcname: infer
{self:<class 'jedi.evaluate.compiled.context.EmptyCompiledName'>
}
jedi.evaluate.compiled.context::infer(369):         return NO_CONTEXTS
jedi.evaluate.compiled.context::infer(369): RETURN <class 'jedi.evaluate.base_context.ContextSet'>
 --- modulename: jedi.common.context, funcname: __iter__
{self:<class 'jedi.evaluate.base_context.ContextSet'>
}
jedi.common.context::__iter__(46):         for element in self._set:
jedi.common.context::__iter__(46): RETURN None
jedi.evaluate.context.instance::infer(336): RETURN None
jedi.common.context::__init__(17):         for context in iterable:
jedi.common.context::__init__(17): RETURN None
jedi.evaluate.base_context::wrapper(434): RETURN <class 'jedi.evaluate.base_context.ContextSet'>
 --- modulename: jedi.common.context, funcname: __iter__
{self:<class 'jedi.evaluate.base_context.ContextSet'>
}
jedi.common.context::__iter__(46):         for element in self._set:
jedi.common.context::__iter__(46): RETURN None
jedi.evaluate.compiled.context::api_type(284): RETURN None
jedi.api.classes::_complete(408): RETURN None
jedi.api.classes::complete(439): RETURN None
jedi.api.completion::filter_names(44): RETURN None
jedi.api.completion::completions(104): RETURN None

Thrameos · 2019-06-23T04:44:24Z

So the specific problem is

    @property
    def api_type(self):
        return next(iter(self.infer())).api_type

iter returned none because it was an EmptyCompiledName and then next was applied to None.

Thrameos · 2019-06-23T04:54:28Z

Changing the line to

    @property
    def api_type(self):
        api = self.infer()
        if not api:
            return "instance"
        return next(iter(api)).api_type

Corrected the problem.

davidhalter · 2019-06-23T16:48:55Z

Hmm, what I don't understand is: How can it even be a EmptyCompiledName. These classes are completely unrelated. The change you made is in CompiledName.

You can also call jedi.set_debug_function(), that might help.

Thrameos · 2019-06-23T20:01:04Z

It seems like jedi magic to me...

jedi.api.classes::_complete(408):         if self._name.api_type == 'param' and self._stack is not None:
 --- modulename: jedi.evaluate.compiled.context, funcname: api_type
{self:<class 'jedi.evaluate.context.instance.CompiledInstanceName'>
}

We start with a CompiledInstanceName

 --- modulename: jedi.evaluate.base_context, funcname: wrapper
{args:(<CompiledInstanceName: (<CompiledContextName: string_name=jpype._jclass>).CASE_INSENSITIVE_ORDER>,), kwargs:{}, func:<class 'function'>
}
jedi.evaluate.base_context::wrapper(434):         return ContextSet(func(*args, **kwargs))

We call ContextSet

 --- modulename: jedi.common.context, funcname: __init__
{self:<class 'jedi.evaluate.base_context.ContextSet'>, iterable:<class 'generator'>
}
jedi.common.context::__init__(16):         self._set = frozenset(iterable)
 --- modulename: jedi.evaluate.context.instance, funcname: infer
{self:<class 'jedi.evaluate.context.instance.CompiledInstanceName'>
}

Still a CompiledInstanceName

jedi.evaluate.context.instance::infer(336):         for result_context in self._class_member_name.infer():
 --- modulename: jedi.evaluate.compiled.context, funcname: infer
{self:<class 'jedi.evaluate.compiled.context.EmptyCompiledName'>
}

The _class_member_name of the object we were working on was an EmptyCompiledName

jedi.evaluate.compiled.context::infer(372):         return NO_CONTEXTS
jedi.evaluate.compiled.context::infer(372): RETURN <class 'jedi.evaluate.base_context.ContextSet'>
 --- modulename: jedi.common.context, funcname: __iter__
{self:<class 'jedi.evaluate.base_context.ContextSet'>
}
jedi.common.context::__iter__(46):         for element in self._set:
jedi.common.context::__iter__(46): RETURN None

Boom we have a bad result flow back to iter.

So how did it get there in the first place?

It originated here...

jedi.evaluate.compiled.context::_get(402):         if is_descriptor or not has_attribute:
jedi.evaluate.compiled.context::_get(403):             return [self._get_cached_name(name, is_empty=True)]
 --- modulename: jedi.cache, funcname: wrapper
{self:<class 'jedi.evaluate.compiled.context.CompiledObjectFilter'>, args:('CASE_INSENSITIVE_ORDER',), kwargs:{'is_empty': True}, method:<class 'function'>
}
jedi.cache::wrapper(137):         cache_dict = self.__dict__.setdefault('_memoize_method_dct', {})
jedi.cache::wrapper(138):         dct = cache_dict.setdefault(method, {})
jedi.cache::wrapper(139):         key = (args, frozenset(kwargs.items()))
jedi.cache::wrapper(140):         try:
jedi.cache::wrapper(141):             return dct[key]
jedi.cache::wrapper(141): EXCEPTION (<class 'KeyError'>, KeyError((('CASE_INSENSITIVE_ORDER',), frozenset({('is_empty', True)}))), <traceback object at 0x7f7a8f999e08>)
jedi.cache::wrapper(142):         except KeyError:
jedi.cache::wrapper(143):             result = method(self, *args, **kwargs)
 --- modulename: jedi.evaluate.compiled.context, funcname: _get_cached_name
{self:<class 'jedi.evaluate.compiled.context.CompiledObjectFilter'>, name:CASE_INSENSITIVE_ORDER, is_empty:True
}
jedi.evaluate.compiled.context::_get_cached_name(411):         if is_empty:
jedi.evaluate.compiled.context::_get_cached_name(412):             return EmptyCompiledName(self._evaluator, name)
 --- modulename: jedi.evaluate.compiled.context, funcname: __init__
{self:<class 'jedi.evaluate.compiled.context.EmptyCompiledName'>, evaluator:<class 'jedi.evaluate.Evaluator'>, name:CASE_INSENSITIVE_ORDER
}
jedi.evaluate.compiled.context::__init__(368):         self.parent_context = evaluator.builtins_module
jedi.evaluate.compiled.context::__init__(369):         self.string_name = name
jedi.evaluate.compiled.context::__init__(369): RETURN None

Then propagated here

jedi.evaluate.context.instance::__init__(331):         self._class = klass
jedi.evaluate.context.instance::__init__(332):         self._class_member_name = name
jedi.evaluate.context.instance::__init__(332): RETURN None
jedi.evaluate.context.instance::<listcomp>(360):             CompiledInstanceName(self._evaluator, self._instance, self._class, n)
jedi.evaluate.context.instance::<listcomp>(361):             for n in names
 --- modulename: jedi.evaluate.context.instance, funcname: __init__
{self:<class 'jedi.evaluate.context.instance.CompiledInstanceName'>, evaluator:<class 'jedi.evaluate.Evaluator'>, instance:<class 'jedi.evaluate.context.instance.CompiledInstance'>, klass:<class 'jedi.evaluate.compiled.context.CompiledObject'>, name:<class 'jedi.evaluate.compiled.context.EmptyCompiledName'>, __class__:<class 'jedi.evaluate.context.instance.CompiledInstanceName'>
}
jedi.evaluate.context.instance::__init__(325):         super(CompiledInstanceName, self).__init__(
jedi.evaluate.context.instance::__init__(326):             evaluator,
jedi.evaluate.context.instance::__init__(327):             klass.parent_context,
jedi.evaluate.context.instance::__init__(328):             name.string_name
 --- modulename: jedi.evaluate.compiled.context, funcname: __init__
{self:<class 'jedi.evaluate.context.instance.CompiledInstanceName'>, evaluator:<class 'jedi.evaluate.Evaluator'>, parent_context:<class 'jedi.evaluate.compiled.context.CompiledObject'>, name:class_
}
jedi.evaluate.compiled.context::__init__(267):         self._evaluator = evaluator
jedi.evaluate.compiled.context::__init__(268):         self.parent_context = parent_context

davidhalter · 2019-06-23T20:31:18Z

Well, ok that was just laziness on my side. I actually didn't properly grep for it and didn't see the CompiledInstanceName subclass.

I guess you can just create a pull request with the change you proposed.

Thrameos · 2019-06-23T21:14:36Z

Okay I have the trace file loaded at ftp://ftp.llnl.gov/outgoing/nelson85/jedi/traceJedi.tar.gz if you would like to examine further. I will put in a pull request as soon as I am free.

jcafhe · 2019-06-24T17:00:10Z

Thanks for your fix, it solves the exact same exception but with PyQT5 objects.

import jedi
source = """
from PyQt5 import QtWidgets
foo = QtWidgets.QFrame()
foo."""
script = jedi.Script(source, 4, 4)
script.completions() # raise a RuntimeError: generator raised StopIteration

However, I'm not used to call jedi directly (using thru spyder 4.0b2), but I guess it may solve #1236 too.

davidhalter mentioned this issue Jun 23, 2019

PyGObject completion fails in vim and deoplete #1345

Closed

Thrameos mentioned this issue Jun 24, 2019

Fix for EmptyCompiledName #1352

Merged

davidhalter closed this as completed in #1352 Jun 24, 2019

jcafhe mentioned this issue Jun 25, 2019

Internal Python Language Server error spyder-ide/spyder#9655

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exception on completion from compiled object using JPype #1351

Exception on completion from compiled object using JPype #1351

Thrameos commented Jun 22, 2019

davidhalter commented Jun 23, 2019

Thrameos commented Jun 23, 2019

Thrameos commented Jun 23, 2019

Thrameos commented Jun 23, 2019

Thrameos commented Jun 23, 2019

Thrameos commented Jun 23, 2019

Thrameos commented Jun 23, 2019

davidhalter commented Jun 23, 2019 •

edited

Loading

Thrameos commented Jun 23, 2019

davidhalter commented Jun 23, 2019

Thrameos commented Jun 23, 2019

jcafhe commented Jun 24, 2019 •

edited

Loading

Exception on completion from compiled object using JPype #1351

Exception on completion from compiled object using JPype #1351

Comments

Thrameos commented Jun 22, 2019

davidhalter commented Jun 23, 2019

Thrameos commented Jun 23, 2019

Thrameos commented Jun 23, 2019

Thrameos commented Jun 23, 2019

Thrameos commented Jun 23, 2019

Thrameos commented Jun 23, 2019

Thrameos commented Jun 23, 2019

davidhalter commented Jun 23, 2019 • edited Loading

Thrameos commented Jun 23, 2019

davidhalter commented Jun 23, 2019

Thrameos commented Jun 23, 2019

jcafhe commented Jun 24, 2019 • edited Loading

davidhalter commented Jun 23, 2019 •

edited

Loading

jcafhe commented Jun 24, 2019 •

edited

Loading