Recurrent Connections in processes exhibit strange behavior on .run() (non-terminating)

[ashishrao7]

A process (can be hierarchical or sub) when connected recurrently doesn't terminate after the process.run() command. The same process can be connected to another process serially and run without problems.

A minimal example for this behavior is detailed below. Two process are created where process 1 has one InPort and one OutPort and process 2 has only an InPort. Initially process 1 is connected to itself and run. This leads to non-terminating behavior. Also, the get() function to inspect the process variable leads to non-terminating behavior. The same process can be connected to process 2 and run without any problems. The problem appears to be recurrent connections for processes. The same behavior was observed for hierarchical process that housed sub-processes with recurrent connections (not shown in code below). The code that demonstrates the behavior I described above is given below. It has been commented to make the issue clear. @joyeshmishra

import numpy as np

from lava.magma.core.process.process import AbstractProcess
from lava.magma.core.process.variable import Var
from lava.magma.core.process.ports.ports import InPort, OutPort
from lava.magma.core.sync.protocols.loihi_protocol import LoihiProtocol
from lava.magma.core.model.py.ports import PyInPort, PyOutPort
from lava.magma.core.model.py.type import LavaPyType
from lava.magma.core.resources import CPU
from lava.magma.core.decorator import implements, requires
from lava.magma.core.model.py.model import PyLoihiProcessModel
from lava.magma.core.model.sub.model import AbstractSubProcessModel
from lava.magma.core.run_conditions import RunSteps
from lava.magma.core.run_configs import Loihi1SimCfg   


class RecurrentProcess(AbstractProcess):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        shape = kwargs.pop("shape", (1,1))
        self.s_in = InPort(shape=shape)      
        self.x = Var(shape=shape, 
                             init=np.zeros(shape)
                            )
        self.a_out = OutPort(shape=shape)


class OutProbeProcess(AbstractProcess):
    def __init__(self, **kwargs):
        """Use to set read output spike from a process

        Kwargs:
            out_shape (int tuple): set OutShape to custom value 
        """
        super().__init__(**kwargs)
        shape = kwargs.pop("out_shape", (1,1))
        self.s_in = InPort(shape=shape)      
        self.spike_out = Var(shape=shape, 
                             init=np.zeros(shape)
                            )

@implements(proc=OutProbeProcess, protocol=LoihiProtocol)
@requires(CPU)
class PyOPPModel(PyLoihiProcessModel):
    s_in: PyInPort = LavaPyType(PyInPort.VEC_DENSE, np.int32, precision=24)
    spike_out: np.ndarray = LavaPyType(np.ndarray, np.int32, precision=24)

    def run_spk(self):
        s_in = self.s_in.recv()
        self.spike_out = s_in
        


@implements(proc=RecurrentProcess, protocol=LoihiProtocol)
@requires(CPU)
class PyRPModel(PyLoihiProcessModel):
    s_in: PyInPort = LavaPyType(PyInPort.VEC_DENSE, np.int32, precision=24)
    x: np.ndarray = LavaPyType(np.ndarray, np.int32, precision=24)
    a_out: PyOutPort = LavaPyType(PyOutPort.VEC_DENSE, np.int32, precision=24)

    def run_spk(self):
        s_in = self.s_in.recv()
        self.x += s_in
        self.a_out.send(self.x)
        #self.a_out.flush()

if __name__ == '__main__':
    input_spike = np.array([[1],[2]])
    rec_process = RecurrentProcess(shape=input_spike.shape)
    # another process made to test serial connection which works well (Program terminates)
    out_spike_process = OutProbeProcess(out_shape=input_spike.shape)

    # Uncomment below line and comment the line after that to see how the process connected with serial processes works without issues (Change Blocking to True)
    
    #rec_process.a_out.connect(out_spike_process.s_in)
    rec_process.a_out.connect(rec_process.s_in)
    
    print("Starting Recurrent Process")
    # Blocking False for recurrrent process and True for serial connection
    rec_process.run(condition=RunSteps(num_steps=1, blocking=True), 
                            run_cfg=Loihi1SimCfg(select_sub_proc_model=False))
    #print(rec_process.vars.x.get())    #Running get() method doesn't go to next line only for recurrent connection
    print("Trying to end Recurrent Process")
    rec_process.stop()
    # recurrent process never goes to end of execution
    print("Process Ended")

[awintel]

Perhaps there's a bug in Lava but the first issue with the code above is this:
def run_spk(self):
s_in = self.s_in.recv()
self.x += s_in
self.a_out.send(self.x)
#self.a_out.flush()

This will block by design in case of a recurrent connection! When run_spk starts executing, it will block on recv() until there is input on s_in .
Where is s_in getting it's data from? From a_out of the same process.
When will a_out send any data to s_in? Two lines later!

In case of a serial connection, this won't happen because the port sending data belongs to a different independent process running in parallel. So nothing will block in this case.

[ashishrao7]

Yeah that seemed to solve the issue. Thanks! Maybe if there's a tutorial on this in the future we state this explicitly as a warning so that it saves time for other people :)

[awintel]

Yes, we should probably create an FAQ with common/possible answers to "Why does my code deadlock?" when writing asynchronous code.