Flatter module hierarchy

[mathisrichter]

I would like to suggest a new module hierarchy to

• expose all user-facing APIs on a high level,
• make import paths shorter, and
• group related content closer together.

We currently have this:

lava
- magma
  - compiler
    - builders
    - channels
    - subcompilers
  - core
    - learning
    - model
    - process
    - sync
  - runtime
- proc
  - conv
  - dense
  - lif
  - ...
- utils
  - dataloader
  - float2fixed.py
  - profiler.py
  - system.py
  - validator.py
  - visualizer.py
  - weightutils.py

This is not fully thought through but maybe something like this:

lava
- core
  - compiler
    - builders
    - channels
    - subcompilers
  - learning
  - process
  - model
  - sync
- neurons
  - lif
  - scif
  - sdn
  - rf
  - ...
- connections
  - conv
  - dense
- sources
  - dataloader
- sinks
  - monitor
- learning
  - learning_dense
  - rules
    - stdp
- datasets (utils/dataloader)
  - mnist
- profiler
- fixed_point (utils/float2fixed plus future developments)

[awintel]

This flatter directory structure looks extremely cluttered to me.
Can't we achieve something like this with init.py files to maintain a logical and hierarchical order of the code but ease imports?
To my knowledge, TF does something similar. They have a top level init.py that pulls together everything a normal user requires for creating procs.
In turn, I don't see an issue with having long import paths in core infrastructure code like the compiler.

[weidel-p]

We can achieve something like this with init files, but it's incompatible to the implicit namespace we are using as the implicit namespace is implemented by the absence of init files.

I would be in favor to move to a normal namespace. Then we could make the hierarchy look flatter and also create "module docstrings" in the init files which would improve our API reference.

[mathisrichter]

With normal namespaces, will we be able to bring together the directory structures from our separate repositories (lava-nc, lava-loihi, libraries) into a single, coherent structure? I believe this is why we are using implicit namespaces.

[mathisrichter]

This flatter directory structure looks extremely cluttered to me.

What makes it look cluttered to you? Is it that the content of the directories is sometimes Processes and sometimes other APIs?

[awintel]

This flatter directory structure looks extremely cluttered to me.

What makes it look cluttered to you? Is it that the content of the directories is sometimes Processes and sometimes other APIs?

Well, the purpose of a hierarchy is that related things are grouped together in a logical way. This makes it at least easier for me to navigate a directory structure and recognize what is what and how things are related.
By flattening things out, this logical structure is obviously lost. Now, "unrelated" things like 'neurons' and 'profiler' are on the same level. There's also nothing specific anymore that we can say is the "standard proc library" (formerly /proc).

[awintel]

Unrelated note:

• learning
• core
  • learning_rule.py
  • product_series.py

This is Loihi internals leaking to the top level.

[mathisrichter]

My idea was to pull up all user-facing APIs to the top layer and put them into modules with descriptive names right under lava. All of the top-level directories have in common that they are elements that users would use or instantiate in an application.
The core architecture of course has to go somewhere, so it would go into core. That, by definition, must be different.

I felt it would be easier for users to find elements they can use just by looking at the code. They would not have to understand what magma is (apparently a source of confusion), and can see that scif seems to be a type of neuron.

I agree with your comment on learning - that was an error on my part. I have now changed it in my proposal, hiding internals in core.

I understand that the different nature of the APIs (some being Processes, others not) may seem confusing. It does not bother me that much personally. I thought I would just start this discussion and others could integrate their thoughts.

[awintel]

What about the alternative idea of having a single top level init file that pulls everything a user commonly uses together? Would give you the same look and feel but not clutter the directory structure. Again... works well for TF:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/__init__.py
There should be a solution to the implicit namespace issue...

I'm not insisting on the name 'magma'. It could be just 'core'. But then we are also not really consistent, as 'core' would combine stuff the user needs and that the user doesn't need on a daily basis.

I'm just concerned with the top-level directory just growing arbitrarily as time moves on. If we don't impose any basic structure where utils, procs, and other stuff that people might invent go, then people will just throw everything at the top level as new utils, backends, peripherals, etc. are added.

I'm the least active developer right now, so maybe my opinion does not count much. I just feel (a) it would already be messy and (b) can only get messier over time.