Tutorial tests stalling and import warning from wrong path construction.

[AjayKumar-LVLS]

I have followed the steps for installation of lava-optimization library and upon pytest command, the execution gets struck at test_tutorial.py indefinitely. The exact place where it is getting struck is shown in the images:

So, I exited from it and tried executing QUBO tutorial problem. I am able to execute the problem in QUBO, but I am getting a warning which says:

/home/ubuntu/lava/src/lava/magma/compiler/compiler_graphs.py:866: UserWarning: Cannot import module '<module 'solver' from '/home/ubuntu/lava-optimization/src/lava/lib/optimization/solvers/generic/solver.py'>' when searching ProcessModels for Process 'StochasticIntegrateAndFire'.
warnings.warn(

Also, I tried to install the packages manually in an virtual environment, (i.e., 1. clone lava and lava-optimization 2. set python path of lava and lava-optimization source 3. pip install the dependencies eg. asteval, numpy, etc.).

I am able to execute the problem in QUBO, but again getting the same warning as before:

/home/ajay/lava/src/lava/magma/compiler/compiler_graphs.py:862: UserWarning: Cannot import module '<module 'solver' from '/home/ajay/lava-optimization/src/lava/lib/optimization/solvers/generic/solver.py'>' when searching ProcessModels for Process 'StochasticIntegrateAndFire'.
warnings.warn(

Is the execution going on as it should be happening and results are good? or the results obtained are erroneous?
Please let me know is there anything that i missed during the installation, so it is showing this warning.

The python path is set on both installations as:
/home/ubuntu/lava/src:/home/ubuntu/lava-optimization/src

Thanks in advance
Ajay

[AjayKumar-LVLS]

The observations found while debugging, If useful for clarifying the issue.

I tried to trace this warning and I was curious to see that the path for to find solver.py file was "lava-optimization.src.lava.lib.optimization.solvers.generic.lava.lib.optimization.solvers.generic.solver.py" instead of "lava-optimization.src.lava.lib.optimization.solvers.generic.solver.py".

I was afraid to see this was due to concatenation happening in this line highlighted as (1) in above screenshot.
Which takes path as "lava-optimization.src.lava.lib.optimization.solvers.generic.lava.lib.optimization.solvers.generic.solver.py".

[GaboFGuerra]

Hi AjayKumar,

Thanks for pointing this out, and inspecting the root cause. I was out so will only now start looking at this issue, will let you know once I push a solution for it ;)

The solver works if you can sucessfully run the TestWorkloads tests here:

lava-optimization/tests/lava/lib/optimization/solvers/generic/test_solver_cpu_backend.py

Line 152 in 3032f6b

class TestWorkloads(unittest.TestCase):

Thanks,
Gabriel

[GaboFGuerra]

@AjayKumar-LVLS I also renamed the issue to better reflect what you are seeing.

[GaboFGuerra]

It seems test_tutorials.py does not stall but instead takes very long to run, @phstratmann approximately how long does the full tutorial takes to run when the user executes it? Maybe we can reduce the scope during the testing.

@AjayKumar-LVLS could you confirm whether this test completes if you allow it for at least 10 minutes? (It takes less on my end, but just in case)

[AjayKumar-LVLS]

HI @GaboFGuerra,

I checked again today executing pytest, test_tutorials.py file did not exit from executing even after 20 mins.

Also, I tried executing the tests under TestWorkloads. I see the same error warnings

"/home/ubuntu/lava-optimization/.venv/lib/python3.8/site-packages/lava/magma/compiler/compiler_graphs.py:862: UserWarning: Cannot import module '<module 'solver' from '/home/ubuntu/lava-optimization/src/lava/lib/optimization/solvers/generic/solver.py'>' when searching ProcessModels for Process 'StochasticIntegrateAndFire'.warnings.warn()."

However, the tests are passed. I hope solver works!!, Please clarify on this!
The execution results screenshot is as pasted below:

Thanks for your response.

[phstratmann]

Thanks @AjayKumar-LVLS for reporting this issue, and for investigating it!

This is definitely not an issue with the time the tutorial needs to run. That should be <<1min.

It seems that there is a bug by now. I'll investigate.

[phstratmann]

The warning is fixed by this lava PR:
lava-nc/lava#418

But that doesn't solve the test_tutorials.py issue. Still investigating.

[AjayKumar-LVLS]

hi @phstratmann ,

I have replaced the lava folder with the current lava library files. I am still getting the same error warning even after replacing the lava changes.
"/home/ubuntu/lava-optimization/.venv/lib/python3.8/site-packages/lava/magma/compiler/compiler_graphs.py:866: UserWarning: Cannot import module '<module 'solver' from '/home/ubuntu/lava-optimization/src/lava/lib/optimization/solvers/generic/solver.py'>' when searching ProcessModels for Process 'StochasticIntegrateAndFire'.
warnings.warn(".

Please let me know if I am missing anything.

[phstratmann]

Hi @AjayKumar-LVLS ,
Can you check whether line 753-755 of your /home/ubuntu/lava-optimization/.venv/lib/python3.8/site-packages/lava/magma/compiler/compiler_graphs.py file reads as follows?
classes = [m[1] for m in inspect.getmembers(module, inspect.isclass) if m[1].__module__ == module.__name__] for cls in classes:
And please also check that you have restarted the ipython kernel in jupyter notebook.

Please confirm if you see this code in your lava repo.

[AjayKumar-LVLS]

Hi @phstratmann

Yeah I see these changes are not reflected in line 753-755 of your /home/ubuntu/lava-optimization/.venv/lib/python3.8/site-packages/lava/magma/compiler/compiler_graphs.py file. I have followed the steps for installation as per the installation process, But these recent changes in lava file is not reflected in re-installation of lava-optimization. I initially saw that the lava files in path
"/home/ubuntu/lava-optimization/.venv/src/lava/" were also not reflected with recent updates. so I updated lava folder here!

Now, I replaced the latest compiler_graphs.py file manually in that path "/home/ubuntu/lava-optimization/.venv/lib/python3.8/site-packages/lava/magma/compiler/" and now I see execution is happening with out the warning.

Thanks for your support and response!

[AjayKumar-LVLS]

Thanks for looking into the posted issue and resolving it!

Any help here on setting hyper-parameters is appreciated

I am facing difficulty setting hyper-parameters to get a good solution. Please share the knowledge on best-known practice followed now to choose good hyper-parameters manually in the example problem "tutorial_02_solving_qubos". It will be of great help for me to set hyper-parameters in similar way.

Also, please share if there are any guidelines to set hyper-parameters manually.

[phstratmann]

Hooray, that's good news! Please ensure that Lava is always up to date, also in your side-packages.

Could you also check if the test_tutorials.py runs now?

Re hyperparameter tuning: We will release a tutorial asap that also discusses hyperparameter tuning. Plan is to present this tutorial at the upcoming workshop.

[AjayKumar-LVLS]

Hi @phstratmann

No the tests stalling now also. I have checked with replaced just compiler_graphs.py file, also the whole lava folder.

I tested for the new changes in lava that reverted the updates, Now it is having error warning with tests stalling.

Thanks for your reply, I understand the team is working hard to release tutorials in the next upgrade. If possible, before that any useful info on best-known practice followed now to choose good hyper-parameters manually in the example problem "tutorial_02_solving_qubos" is much appreciated for me to execute other qubo problem on which I am working.

[phstratmann]

Dear @AjayKumar-LVLS ,
Thanks for your patience, as we were preparing the release and the workshop.

As you have experienced, my bug fix was reverted initially. The reason was that it changed the default ProcModels, and thus might have changed the default behavior for users who did not specify their ProcModels in detail.

With the release from last week, the warning and the stalling unit tests should be fixed. Can you confirm?

[phstratmann]

Re: Parameter tuning

At the moment, we are continuously improving the performance of the QUBO solver. As part of this, we expect changes to the solver dynamics and hyperparameters. As soon as we have verified the solver for increasingly large workloads, we can release more guidance for the hyperparameter tuning. That will probably include an analytic analysis of the hyperparameters search.

For now, I can provide you with more information about how the solver works under the hood. Based on this, you may be able to infer the best hyperparameters.
In particular, a QUBO problem is implemented in the following way:

If you can think of ways to tune hyperparameter, or to improve the solver dynamics, then we are always appreciating any feedback!

[AjayKumar-LVLS]

Hi @phstratmann

Thanks for you reply!

The execution is happening without the warning. However, the tests execution is still taking time.

[AjayKumar-LVLS]

Hi @phstratmann

Sorry for late reply!

Hyper-parameters tuning- If there is a relation between target cost and search space for step size and steps to fire. Then, It may be helpful, given what range of values for step size and steps to fire are suitable for a given target cost range in form of table/map.

For automating the hyper-parameters I did a quick literature search and found this reference-

1. Bayesian Multi-objective Hyperparameter Optimization for Accurate, Fast, and Efficient Neural Network Accelerator Design.
2. Neuromorphic Hardware Learns to Learn - https://www.frontiersin.org/articles/10.3389/fnins.2019.00483/full

Thanks

[GaboFGuerra]

Thanks @AjayKumar-LVLS and @phstratmann for digging on the issue. Is this resolved with the releases that have happened? Or is the stalling still happening? @phstratmann did you ever reproduce the stalling? or is this something that has only happened on @AjayKumar-LVLS's end?