Events: System wide event queue to replace current broadcast.

[jsimnz]

Part of #822

Currently we have a p2p-specific implementation for the broadcaster to announce new CRDT log events so the p2p system can publish the update over the network.

This can be generalized to work over the entire DB system, and include all kinds of events beyond new CRDT logs. Eg. More specific updates, create-log, update-log, create-collection, create-index, index-online (index is ready to be used and has been backfilled/updated to current state), new-peer, etc.

Moreover, this needs to concretely fix #629 so there is no race condition between the broadcaster and the consumer of the event queue.

This is a result of how the transactions work due to our ACID requirements. Data isn't officially in the DB until the transaction as been succesfully commited. However, the current design initiates the broadcast before the transaction is commited, and the consumer likely doesn't have access to the transaction so they access the DB directly to get the infromation required. But, the tx hasn't commited, so the data associated with the event doesn't yet exist as commited information on the DB.

The likely way out of this is to rely on a system already present in Defra, which is Transaction Callbacks (error & success). Which allow us to register functions to be executed on succesful commits or errors of a transaction. So we can link our event queue publish action to the tx success callback.

Something to explore.

[jsimnz]

Look into unifying with libp2p.Host.EventBus

[orpheuslummis]

problems to avoid:

• over-serialization & blocking
• backpressure from the event system badly impact other parts, e.g. activity in p2p affecting db

[AndrewSisley]

Would be nice if this can also be used to remove the P2P/broadcasting stuff from the db package too IMO

[AndrewSisley]

Thought: We might also want to consider the following:

• Persistence, should the event queue be wiped if the database goes offline? (should probs be configurable)
• Persistence part 2, the capacity of the queue and how long the items remain in it should probably be configurable.
• Exposure, it could be really handy to have the queue exposed as part of our public abi
• Security/privacy - for the future, all items in the queue should probably not be exposed to everyone, but as the protocol side of defra is not up and running yet, this is not something we can actively code for really atm I think

[jsimnz]

Would be nice if this can also be used to remove the P2P/broadcasting stuff from the db package too IMO

What do you mean? ATM, theres no P2P relation in the db package?

The "broadcaster" in the db package is a local one, that is a half-hazard attempt of a DB wide event system. So its likely that any format the solution to this issue of a event system takes, will still have a similar relationship in the DB package that the current "broadcaster" has.

The name "broadcaster" of the current setup is a bit of red-herring as it suggests actualy networking stuff, but its local events only

[AndrewSisley]

I meant in the merkle package, I thought it was in db

[jsimnz]

It exists in both :)

Merkle:

defradb/merkle/crdt/merklecrdt.go

Line 87 in bbb9b42

func (base *baseMerkleCRDT) Broadcast(ctx context.Context, nd ipld.Node, delta core.Delta) error {

DB:

defradb/db/db.go

Line 62 in bbb9b42

broadcaster corenet.Broadcaster