-
Notifications
You must be signed in to change notification settings - Fork 890
Media Driver Operation
This section assumes familiarity with Aeron Terminology.
The media driver is a separate process that provides a means to demux and provide buffers of data for Aeron to process from various transmission media. It decouples the means of data transmission from protocol processing. There can be many different media drivers for various media, to take advantage of different optimisations (such as Linux epoll and sendmmsg/recvmmsg).
The API/stack (or just API) is the Aeron library that interacts with the application and provides the API as well as the bulk of the protocol processing. The media driver and the API/stack communicate via a set of shared memory buffers.
Each Term (which is unique to a Session and unique to a Channel) is associated with a buffer. This is purely for data.
For receiving data, the media driver receives off a socket and must look up Session ID, Channel ID, and Term ID and send that data to a specific buffer.
For sending data, the buffer contains data to be sent to a given socket or communication endpoint. Thus the media driver picks up the data and just sends it as there is an easy/direct association between buffer and socket or communication endpoint.
There are two control buffers between the API/stack and the media driver. One is for the API/stack to send control Frames and instructions to the media driver. The other is for received control Frames and notifications to the media driver. These buffers imply no ordering. Events are events and order is not assumed. Loss recovery is not performed on these buffers either.
This means that at any one time, the number of shared memory buffers is:
- 1 per Term ID for receive (incoming data)
- 1 per Term ID for send (outgoing data)
- 1 per media driver for control (incoming to API/stack)
- 1 per media driver for control (outgoing from API/stack)
How do send buffers get created?
The API creates a new buffer for a new Term ID. This is regardless of old Session and Channel or new Session and Channel.
How do receive buffers get created?
The API explicitly desires data on a given Channel ID. At this point, it does not know about what sources (Session IDs) and Term IDs will appear. But it does know that the API is interested in a specific Channel ID. The API creates a new buffer for the initial Term ID it will begin with. When data arrives on that Channel ID, the Session ID and Term ID are filled in for that buffer. The media driver will create new buffers for Terms as needed on this Channel ID. If a new Session ID is seen, it is handled as a new Term would be and the media driver creates it.
When a new send or receive buffer is created, a control message for media driver <—> API/stack must be sent to let the other side know of the new channel and act appropriately.
A typical media driver has 3 threads (agents) which can be scheduled on 1 (SHARED), 2 (SHARED_NETWORK), or 3 (DEDICATED) OS threads depending on selected threading mode.
Conductor Thread:
- This reads from the incoming control buffer setting up and tearing down new sessions, channels and receivers. This thread also deals with detecting loss, handling NAK state, and retransmitting based on received NAKs.
- Manages the allocation of buffers.
- Sends errors or acknowledgement messages on the outgoing control buffer.
- Maintains internal mapping of term buffers, and sends information about new term buffers and their mapping to the receiver and sender threads.
- Looks for loss in all Term buffers on the receive side
- Looks for NAKs it needs to retransmit for, grab data and send retransmit
- Tracks recent retransmits and ignores new NAKs.
Receiver Thread:
- Reads data and control messages from the sockets.
- Forwards along data onto the receiver buffer that corresponds to the Channel ID/Session ID/Term ID triple.
- Receives SMs for Senders, needs to map SessionID/ChannelID/TermID to a sender thread Term buffer and update flow control state for Sender Thread.
- Receives NAKs for Senders, needs to map SessionID/ChannelID/TermID to a sender thread Term buffer and update NAK handling state for Admin Thread.
Sender Thread:
- Reads data buffers from producers and sends data out on sockets.
- If it can see data to send it maps from Term buffer to send socket. This is setup by AdminThread.
- It is informed of new Term buffers by Admin Thread.
- Tracks per ChannelID flow control state, updated by Receiver Thread when it receives an SM.
Communication between threads is in the form of MPSC ring buffers. These are the same data structures used for the control and data buffers between the core and the media driver but use in process byte buffers for memory rather than shared memory IPC.
Suggestion of having the end of a Term have a flag point to the next Term. This can work in both API -> media driver and media driver -> API directions for Term rollover.
The media driver and API will need to know about new Term buffers. New Session IDs, Channel IDs, etc. are, in essence, just new Term buffers.
Data reception is probably the more complex case. New Terms need to be created when new Sources (i.e. new Session IDs) are seen with interesting Channel IDs. I think we have to have the ability for the media driver to create new Term buffers and the API to notice them.
Operation: media driver controls buffer creation for data reception
- media driver is instructed by the API what it is interested in. This includes Channel IDs and Transmission Media specifics (such as IP addresses and ports to listen on for UDP).
- media driver discards any data that does not match. e.g. an uninteresting Channel ID.
- media driver creates new Term buffers as needed.
- media driver signals new Terms to the API via Term rollover flags, Term index buffer, etc.
Data sending is easier. The API creates static Terms for each Channel it will send on. When they have to roll to new Terms, the media driver will know via the rollover flag.
The API and the media driver communicate with one another via the control buffer. Here is a list of commands that need to be exchangeable between the API and media driver.
| Message | Parameters | Description |
|---|---|---|
| Add Receiver | Destination String, List of Channel IDs | Have media driver set up state and listen to a destination and a given set of Channel IDs. Generates a Location Response by the media driver. |
| Add Channel | Destination String, Session ID, Channel ID | Have media driver set up state to be able to send on a Channel within a Session. Generates a Location Response by the media driver. |
| Remove Receiver | Destination String, List of Channel IDs | Have media driver stop listening to a given destination string for the API. Removes all Channel IDs and Terms as well. |
| Remove Source | Destination String, Session ID | Have media driver tear down state and forget about sending on a given Session, its Channels, and its Terms totally |
| Remove Channel | Session ID, Channel ID | Have media driver tear down state and forget about sending on a given Channel and its Terms totally. But keep Session viable. |
| Remove Term | Session ID, Channel ID, Term ID | Have media driver stop allowing a given Term ID to be recoverable. Once created by the API, Terms stay around until they are removed by the API. |
| Request Term | Session ID, Channel ID, Term ID, Destination String | Media driver should setup state for the next Term Buffer |
| Message | Parameters | Description |
|---|---|---|
| Error Response | Code (int), String | Error Response to previous command from API. Indicates command can't be fulfilled. |
| Error Notification | Code (int), String | Notification of an error condition in the media driver to the API |
| New Receive Buffer Notification | Session ID, Channel ID, Term ID, Destination String | Notifications of new buffer(s) due to new Sessions/Sources for receiving data |
| New Sender Buffer Notification | Session ID, Channel ID, Term ID, Destination String | Notifications of new buffer(s) due to new Sessions/Sources for receiving data |
The Core Client Stack also spins up an admin thread which communicates with the admin thread in the media driver via the control protocol. The Receiver and Sender integrate into the existing application's threading model so don't run on their own threads. The receiver and sender communicate with the admin thread by sending messages to its command buffer.
Client Conductor Thread:
- Receives information about how fast data is being written into channels via the command buffer. It then calculates the rate and remaining time before a term buffer is exhausted. When the term buffer is expected to be exhausted soon the admin thread sends out a message to request a new term buffer across its outgoing control buffer.
- Receives responses when new term buffers are mapped by the media driver. These are then communicated to the receiver and sender.
We can rendezvous the new term ids to the receiver and sender by hashing into an array of term buffers which are updated.