Stdlib architecture for ruby

[powerjg]

Originally by Gabriel Busnot on Jira (https://gem5.atlassian.net/browse/GEM5-1333)

I would like to start a discussion about the architecture of the stdlib regarding ruby components (Does such discussion already exist BTW?). I think the current approach lacks some abstractions and makes more complex systems difficult to implement.

Ruby components

I would list the following components as the parts of a ruby system:

1. The ruby system as a whole
2. The controllers (the units implementing the protocol)
3. The transport (refereed to as Network in gem5)
4. The topology

In addition to these concrete entities, I would add the notion of node which is a tightly coupled set of controllers, usually connected with a crossbar. A node can typically be a cluster of 1 or more CPUs with their local caches and a single port to the main transport network.

The current architecture of the stdlib (talking about the ruby part only) is unable to mix these abstractions in a satisfying way (to me 😄).

Current architecture and limitations

Currently, the stdlib architecture relies the following abstractions:

1. The ruby system as a specialization of AbstractCacheHierarchy
2. The controllers as derived instances of the corresponding python SimObjects.
3. The network and its topology as an instance of a concrete class derived from the SimObject Network class.

The AbstractCacheHierarchy instantiates all the controllers, the network, passes the former to the later and is done with it. The network implements the topology defined by the connectControllers method.

I identify the two following main limitations:

1. Nodes do not exist and cannot be understood by the Network class anyway. That works for Crossbar or Pt2Pt but not so well for Mesh.
2. The network becomes the topology while they were previously decoupled (Defining Pt2Pt and Crossbar for SimpleNetwork and Garnet would result in 4 very similar classes)

Proposed architecture

I suggest organizing the ruby stdlib in a way similar to what the CHI protocol did in CHI-config.py with some extra refinements. It is a very rough outline that would need to be translated into some proper interfaces.

Separating back the topology from the network

The topology would be defined as an algorithm describing which nodes/routers to connect with each other.

The network is a top level class providing at least these services:

1. An interface for the topology to instantiate the network components (routers, links, etc.)
2. Top level SimObject for those components

As a result, the topology would work on a collection of nodes and build the network using the network class interface that would include, but not limited to:

1. Create an entry point for each node (could involve instantiating a crossbar if the node includes multiple controllers). An entry point is an abstraction that can be connected to a router.
2. Instantiate a router
3. Connect an entry point to a router and vice versa (ExtLink, could involve instantiating an interface)
4. Connect a router to a router (IntLink)

Wrapping controllers in nodes

Nodes would provide the controller-side services required by the topology algorithms and the Network to operate:

1. Vnets used by the controllers in the node
2. is Local crossbar required (e.g., if the node contains multiple controllers)
3. Optional placement data (e.g., mesh location)

Remarks and suggestions are very welcome!

[powerjg]

I agree with your limitations, and I also strongly agree that this interface can be improved!

Here are my thoughts...

An explicit non-goal: Enabling the cross product of [all protocols] x [all networks] x [all topologies] to work together.

Instead, I want to see cache hierarchies which are a tight coupling of specific instances of a protocol, a network, and a topology.

I definitely like the idea of nodes. I think it's really hard to figure out a way to do this that is generalizable to any protocols/topology. I think that to implement the idea of nodes, but I'm struggling to see how it will work without being part of a specific specialized cache hierarchy.

I would love to see a proof-of-concept based on the ideas here. If there's a way to make nodes both expressive enough and easy enough to use, it could be a great way forward.

[MeatBoy106]

Can you develop why you don't want to target such cross product but prefer a tight coupling of a specific version of each component?

My view is that a network class knows how to connect nodes and switches, a topology knows which node/router to connect with each other and a node knows how to interface one or more ruby controllers from a specific protocol with a network class (provides required properties). The nodes can also provide metadata to the topology algorithm to drive it (e.g. mesh coordinates).

At least, that architecture would provide a working basis for most combinations of upstream protocol/network/topology.

[powerjg]

Yeah... I know that's how the SimObjects are designed. And this flexibility is incredibly powerful!

However, it's a maintenance nightmare. For instance, in the old code, "mesh-dir-corners" would only work with protocols that had a specific number of directories and cores. There was no way for the user to know this, so there were innumerable questions about why things didn't work.

In the stdlib, I want to see that everything "just works" if it's supported. We'll never say that you have to use our stdlib (you could build a different python library on top of gem5's SimObjects). However, for things that are part of gem5's "standard" library of components, I want them all to work "as expected."

My number 1 thing I want to avoid is having to answer questions/issues that say "why doesn't x, y, z work?" with "well, it was never tested and we have no way to recreate your problem." With Ruby (as it is using se.py/fs.py) this is the only possible answer to the questions. Well, I guess you could also answer "come do an internship with me and I'll explain everything to you," but that's also not satisfying.

All this said, I'm very open to a proof-of-concept implementation. If you can demonstrate that this idea works generically and that it is both maintainable and explainable, I would strongly approve! If there is a way to expose all of this configurability to users without making in a maintenance nightmare that would be amazing. So far, I don't have the vision for this myself 😄.

[MeatBoy106]

I see your point and I fully agree. I was not really aiming for an "it just works" solution at that level. I think that it is really nice to have "It just works" whole cache hierarchies available. But it would also be nice to have a nice set of interfaces to define new ruby-based cache hierarchies. I don't think it is possible to achieve a one-size-fits-all solution when it comes to ruby. I am more interested in a "specializes" gracefully architecture.

When I find time to release the network model I use, I'll also provide an stdlib package that follows the design I described above.

[powerjg]

Cool! I am fully on board with a "specializes gracefully" architecture! It sounds like we're on the same page here, now 😄